Know what you are comparing
A training-serving comparison asks whether current production features differ from the training baseline. An inference drift comparison asks whether production features have changed over time. Concept drift describes a change in the relationship between inputs and the target: for example, the same customer behavior may no longer predict churn after a competitor launches a new offer. An input shift can occur without a change in that relationship, and a change in the relationship can occur without a visible input shift.
A currency conversion bug changes all prices supplied to a model. Identify the likely training-serving mismatch and explain why retraining on the erroneous prices would not fix the upstream defect.
Build useful baselines and alerts
Keep a versioned reference for the model and its training data, including preprocessing definitions and feature schemas. Compare like with like: the same feature, unit, transformation, and population. Monitor missing values, unexpected categories, and distributions alongside service latency and errors. Segment by meaningful populations, account for seasonality, and set thresholds using historical variation and the cost of an alert. Distribution distance is a signal, not a direct measure of model accuracy.
A weekday fraud model is busiest on Fridays. Choose a comparison window that respects weekly seasonality, and explain how a seven-day aggregate might hide a sudden one-day pipeline failure.
Triage, validate, and decide
First verify the alert and investigate schema changes, missing data, feature freshness, upstream releases, and serving transformations. Compare predictions and operational metrics to a known baseline. When labels arrive, measure outcome quality and relevant slices on a representative cohort; account for delayed or selectively observed labels. Fix a data defect at its source. If the environment has genuinely changed, evaluate a candidate on recent labeled data and a separate holdout before promotion. Record the evidence and rollback plan.
A lender observes input drift after expanding into a new region. List one benign explanation, one data-pipeline failure, and one potential model-quality concern. State which can be investigated before labels arrive and which requires outcome evidence.
Map the decision to Google Cloud
Google Cloud Model Monitoring documents comparisons between training and production feature distributions and between production windows. Its feature monitoring can surface changes in numerical and categorical inputs. It does not remove the need for label-based quality evaluation, security review, and operational ownership. Consult the current product documentation before setting up a monitoring job: supported features and configuration vary by model and monitoring version.
Sketch a runbook with an owner, alert signal, comparison baseline, first diagnostic query, label-lag expectation, and rollback trigger. Explain how you would distinguish a broken feature pipeline from a real change in customer behavior.
Production feature distributions change after a new region launches, but the outcome labels have not arrived. What is the strongest conclusion?
Keep exploring
- Practice: investigate two monitoring alerts ↗
- Model Monitoring overview ↗
- Monitor feature skew and drift ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.