Machine learning engineers and data scientists spend most of their time testing and validating their models’ performance. But as machine learning products become more integral to our daily lives, the importance of building rigorous testing pipelines for model predictions will only increase. Current ML evaluation techniques are falling short in their attempts to describe the full picture of model performance. Evaluating ML models by only using global metrics (like accuracy or F1 score) produces a low-resolution picture of a model’s performance and fails to describe the model performance across types of cases, attributes, scenarios. It is rapidly becoming vital for ML teams to have a full understanding of when and how their models fail and to track these cases across different model versions to be able to identify regression. We’ve seen great results from teams implementing unit and regression testing techniques in their model testing. In this talk, we’ll cover why systematic unit testing is important and how to effectively test ML models' behavior.

Current ML model evaluation techniques are falling short. Model evaluation using only global metrics (like accuracy or F1 score) produces a low-resolution picture of a model’s performance and fails to meaningfully describe its capabilities across different subsets and scenarios. It is vital for ML teams to have a full understanding of when and how their models fail and to track these behaviors across different model versions to identify improvements and regressions. In this talk, we’ll cover why systematic model testing is important and how to effectively test ML product behavior. Gordon co-founded Kolena, after being burned one too many times by unexpected model performance in mission-critical production scenarios, to build out an ML testing and evaluation platform that tells you what you need to know before your model hits the real world.