Evals 0/1

Evals

Production monitoring

Watching a deployed model for covariate shift and concept drift, and trying a replacement on shadow traffic before it serves anyone. A model that passed its evaluation can still degrade as the data changes.

0 / 4 solved

Reading

  1. Hidden Technical Debt in Machine Learning Systems(opens in a new tab)

    Sculley et al. 2015 NeurIPS 2015 40 min

    The maintenance costs an ML system builds up outside its model code: entangled features (changing anything changes everything), hidden feedback loops, undeclared consumers of predictions, unstable data dependencies, glue code, pipeline jungles and configuration. Its best-known figure draws the model code as a small box inside the serving, data and monitoring infrastructure around it.

  2. Rules of Machine Learning: Best Practices for ML Engineering(opens in a new tab)

    Zinkevich 2016 Essay 60 min

    43 rules from Google's experience building ML products, ordered by project stage, starting with launching without machine learning, keeping the first model simple while getting the pipeline right, and measuring training-serving skew.