Calculate the effect of prevalence
Suppose 10,000 transactions contain 100 actual frauds. A classifier catches 80 of them and flags 198 legitimate transactions. Its recall is 80%, its false positive rate is 198/9,900 = 2%, and its precision is 80/278, approximately 28.8%. A seemingly small false positive rate can still create many alerts when negatives are common. Accuracy is 9,782/10,000 = 97.82%, but it does not describe the review burden or the 20 missed frauds.
Compute the four confusion-matrix cells, total alerts, precision, and recall. If each alert takes three minutes to review, 278 alerts require 834 minutes of review before accounting for staffing overhead.
Separate ranking from the operating point
A score ranks examples, while a threshold determines which ones are acted on. ROC curves compare true positive and false positive rates across thresholds. Precision-recall curves show precision and recall and are often useful when positives are rare; precision depends on prevalence, so compare results on representative populations. A high area under a curve does not automatically mean a particular threshold meets a capacity or cost constraint. BigQuery ML’s ML.ROC_CURVE reports counts and rates across thresholds for supported binary models.
A review team can inspect at most 100 alerts per day. Describe how you would select a threshold using a representative validation set, then test the fixed threshold on a later holdout. Explain why a threshold meeting capacity in one period may exceed it after traffic changes.
Check probability calibration
A calibrated probability has a frequency interpretation over comparable cases: among many cases assigned roughly 0.2, roughly 20% should be positive if calibration holds for that group and period. Ranking and calibration are distinct: a model can order cases well while systematically overestimating probabilities. Examine reliability by score range and relevant slice, with enough examples and uncertainty estimates. If you fit a calibration mapping, fit it on data separate from the model’s training set and assess it on another holdout.
Among 200 cases assigned scores near 0.2, 20 are positive. The observed rate is 10%, not 20%. Describe one possible calibration issue and why a single small group cannot establish performance for every score range or future period.
Account for costs and delayed feedback
Choose an operating rule using the action costs, benefits, review capacity, and expected downstream behavior. If labels arrive late or only reviewed cases receive reliable labels, observed performance can be biased. Track the selection process and obtain representative outcome evidence where feasible. Reassess calibration and thresholds when prevalence, costs, model versions, or review practices change; document the evaluation window and the population to which the decision applies.
A fraud team labels only transactions the model flags. Explain why measured precision on those cases does not reveal how many frauds were missed among unflagged transactions. Propose a carefully governed sampling strategy for additional evaluation.
A model detects 80 of 100 frauds and incorrectly flags 198 of 9,900 legitimate transactions. Approximately what fraction of its alerts are fraud?
Keep exploring
- BigQuery ML threshold metrics ↗
- ROC and precision-recall curves ↗
- Classification metrics and tradeoffs ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.