Start with the four outcomes
A classifier produces true positives, false positives, true negatives, and false negatives. Imagine 1,000 applications with 100 actual fraud cases. At one threshold, the model flags 150: 80 are fraud and 70 are legitimate. That leaves 20 missed fraud cases and 830 correctly cleared applications. Accuracy is 91%, but it hides both the 20 missed cases and the 70 legitimate customers affected. Always name the positive class and the action taken after a positive prediction.
Reconstruct the four cells of this confusion matrix. If a missed fraud costs £100 and an unnecessary review costs £10, what is the total cost? Answer: £2,700, assuming those are the only costs.
Trade precision against recall
Precision is true positives divided by all predicted positives: 80 / 150, or about 53%. Recall is true positives divided by actual positives: 80 / 100, or 80%. High recall finds more positives, while high precision reduces wasted interventions. Neither is universally best. A review team with limited capacity might optimize recall subject to a maximum number of alerts; a safety-critical case might prioritize recall with a minimum acceptable precision.
A second model finds 90 fraud cases but flags 300 applications. Its precision is 30% and recall is 90%. Calculate its cost using the previous assumptions: £1,000 for missed fraud plus £2,100 for reviews, or £3,100. The higher-recall model is more expensive under this specific cost model.
Select a threshold without contaminating your test
A probabilistic model outputs scores; a threshold turns scores into decisions. Lowering the threshold usually increases recall and increases the number of flagged examples. Use a validation set to compare thresholds against costs and operational capacity, then evaluate the chosen model and threshold once on an untouched, representative test set. Check important slices and calibration, and revisit the threshold when prevalence or intervention costs change. ROC AUC summarizes ranking across thresholds; it does not select an operating point or measure actual business cost.
Use Google’s BigQuery ML classification tutorial to train a logistic regression baseline. Evaluate on a held-out dataset, inspect precision and recall, and compare operating thresholds. The tutorial uses billable Google Cloud services; review current pricing and clean up created resources afterward. Documentation: https://docs.cloud.google.com/bigquery/docs/create-machine-learning-model
A model detects 80 of 100 fraud cases and flags 150 cases in total. What are its precision and recall?
Keep exploring
- Practice: offline threshold lab ↗
- BigQuery ML classification tutorial ↗
- BigQuery ML model evaluation ↗
Use the official certification page and its linked resources to validate your coverage. For guided labs, explore the Google Skills learning path. Revisit the practical exercise and explain your decision in your own words before moving on.