Define the label and its maturity
A label is a measurement of an outcome, not automatically the outcome itself. Specify the decision time, outcome window, data source, definition, and delay before a label is considered complete. For a 30-day churn target, customers observed for only five days cannot safely be labeled as non-churners. Corrections and late reports may require extra maturation time. Preserve the definition and version alongside the dataset so later experiments can interpret changes.
You score customers on 1 June and label churn within 30 days. Explain why an extract taken on 10 June contains incomplete labels and how you would choose a later extraction date if reports can arrive up to seven days late.
Look for outcomes you never observe
A fraud system may investigate only flagged transactions; a lending model may observe repayment only for approved applicants. Metrics computed only on those observed outcomes describe the selected population, not necessarily everyone receiving a prediction. Document the selection mechanism, missing-label rate, and affected slices. Where appropriate and ethically governed, collect additional labels through a representative audit sample. Sampling or weighting cannot recover reliable estimates for populations with no observed labels without additional assumptions or data.
A team reviews all flagged transactions and none of the unflagged ones. Explain why it can estimate precision among reviewed alerts but cannot directly count all false negatives. Propose a controlled sample of unflagged cases for audit.
Measure label quality and disagreement
Write clear annotation instructions, examples, and an escalation path for ambiguous cases. Use independent reviewers on a sample, inspect disagreements, and adjudicate without assuming one reviewer is always correct. Track label version, annotator guidance, and quality across languages or other important slices. Human labels can contain mistakes or subjective disagreement; machine-generated labels also need validation against appropriate human review.
Two reviewers agree on 90 of 100 cases. Calculate observed agreement as 90%, then explain why this alone does not reveal which labels are correct or whether agreement is high on a rare critical class.
Split for the intended deployment
Choose splits that reflect how the model will encounter data: chronological splits for future decisions, entity-level separation when generalization to new entities matters, and stratification when useful without violating time or grouping constraints. Fit preprocessing only on training data. If training oversamples rare cases, evaluate on a representative holdout and interpret prevalence-sensitive metrics accordingly. Record sample sizes and uncertainty for small slices and retain a final untouched test set.
A dataset contains repeated visits from the same customer and a rare positive outcome. Design a split for predicting future visits from existing customers, then explain how you would also evaluate performance on entirely new customers.
A fraud model receives confirmed labels only for transactions it flagged. What can the team conclude directly from those labels?
Keep exploring
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.