Put counts beside percentages
A metric is an estimate for the evaluated sample, not a guarantee for every future population. If a model detects 9 of 10 positive cases, recall is 90%; one additional miss would change it to 80%. A larger representative sample generally makes sampling uncertainty smaller, but a large biased or selectively labeled sample can still mislead. Record the numerator, denominator, population, time period, threshold, and label maturity with the metric.
Model A finds 9 of 10 positives; model B finds 900 of 1,000. Both report 90% recall. Explain why the first estimate is more sensitive to one case and why the second still needs a representative sample.
Compare candidates on the same examples
Use the same held-out cohort, label definition, and operating constraints when comparing models. A paired comparison can inspect where one model is right and the other is wrong on the same examples; evaluating on different populations confounds model and data differences. Account for dependence: multiple observations from one customer or neighboring time periods may not be independent. Use an analysis method appropriate to the sampling unit and document uncertainty rather than assuming every row adds independent evidence.
Two models are evaluated on 100 identical cases. They disagree on 12: A is correct on 7 and B on 5. What additional information about sampling and business impact would you need before declaring A the winner?
Inspect slices without losing context
An overall score can hide poor performance on a group, route, language, or time period. Report slice counts and uncertainty alongside metrics, and distinguish predefined critical slices from exploratory findings. Small slices can be volatile; collecting more appropriate data or escalating for review may be more useful than asserting a precise ranking. Looking at many slices also increases the chance of finding an apparent difference by chance, so confirm important findings on fresh data.
A critical slice has 4 positives and the model finds 3. Report recall as 75% with its count, then describe why one additional missed case would change the estimate substantially.
Protect the final evaluation
Use validation data to choose models, thresholds, and features, and reserve a final test set for the chosen system. Repeatedly selecting changes based on the same test set can make it an implicit validation set. When the population or product changes, refresh evaluation data and record which decisions each set influenced. Combine statistical evidence with operational, fairness, safety, and cost gates; a small average gain does not automatically justify release.
A team compares 50 candidates on one test set and deploys the highest-scoring model. Explain why the reported score may be optimistic and propose a better separation of development and final evaluation.
A model detects 9 of 10 positive cases in a small slice. What should accompany the 90% recall figure?
Keep exploring
- Evaluation sample size and imbalanced data ↗
- Training, validation, and test sets ↗
- Model evaluation quality guidelines ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.