Separate operational rollout from impact measurement
A canary limits exposure while checking that a candidate service behaves safely and reliably. An A/B experiment compares outcomes for randomly assigned groups under a defined hypothesis. They can be combined, but a successful canary does not establish that the model improves the business outcome. A shadow deployment can compare predictions without using them for decisions, but cannot by itself measure the effect of acting on those predictions.
A new recommendation model passes a small canary with low error rates. Explain what that establishes and what additional experiment is needed to estimate its effect on customer outcomes.
Choose an assignment unit and primary metric
Define eligibility, a stable randomization unit, treatment and control, the outcome window, and a primary metric before launch. If a customer has multiple sessions, assign consistently at the customer level when cross-session contamination matters. Ensure logging captures assignment, exposure, model version, and outcome with appropriate privacy controls. Check that both groups are comparable and that the observed allocation matches the intended split; investigate discrepancies before interpreting results.
An experiment assigns every request independently, so one customer alternates between two recommendation models. Describe a possible problem and propose a stable unit of assignment.
Plan sample size, timing, and guardrails
Estimate the sample size and duration needed to detect a practically meaningful effect, considering baseline variability, desired power, assignment unit, and business cycles. Predefine operational and safety guardrails, escalation, and a rollback rule. Delayed outcomes require enough time to mature. Repeatedly checking an ordinary fixed-horizon significance test and stopping when it first looks favorable can distort false-positive rates; use a predeclared analysis plan or a valid sequential method.
Control has 100 conversions among 1,000 assigned users and treatment has 110 among 1,000. The observed rates are 10% and 11%: a one-percentage-point absolute difference and a 10% relative lift. Explain why those point estimates alone do not establish a reliable improvement.
Interpret outcomes with their limitations
Analyze the primary metric with uncertainty and review guardrails and relevant slices. Preserve the assignment population in the primary analysis unless the prespecified design justifies another approach; conditioning only on users affected by the treatment can bias a comparison. Investigate missing or delayed outcomes, logging failures, and interference between users. Document the decision, model and feature versions, analysis window, and conditions under which the result may not generalize.
Treatment increases clicks but also increases complaints, and the complaint outcome arrives two weeks late. Describe what evidence is available immediately, which decision must wait for mature outcomes, and what guardrail could stop the rollout early.
A candidate model has low errors during a small canary. What can you conclude?
Keep exploring
- Canary and online experimentation guidance ↗
- Online experiments in MLOps ↗
- Randomized A/B experiments ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.