1. Run the exercise
Download threshold_lab.py. With Python 3 installed, run the following in the directory containing the file. It uses only the Python standard library and makes no network calls.
python threshold_lab.pyThe validation set has eight examples. The script chooses the threshold with the highest recall among candidates that produce no more than four validation alerts; if recall ties, it chooses the higher threshold. A score equal to the threshold is treated as positive. The separate test set is then evaluated once using the selected threshold.
2. Check the result by hand
The selected validation threshold is 0.65. Four examples are flagged, including two of the four actual positives, so validation recall is 0.5. On the separate six-example test set, the counts are TP = 2, FP = 1, TN = 2, and FN = 1. Test precision and recall are both 2/3, approximately 0.667.
Using the same examples to pick a threshold and report its final performance can make the result optimistic. A realistic assessment needs substantially more data, appropriate time or entity splits, and enough outcomes for important slices.
3. Extend the experiment
- Change validation capacity from four to two. Which threshold is selected, and how does recall change?
- Set a threshold above every score. Why is precision undefined when there are no positive predictions?
- Add a test example exactly equal to the selected threshold. Verify the documented inclusive comparison.
- Imagine the number of daily examples doubles. Explain why a fixed threshold that met capacity on this tiny validation set may no longer meet capacity.
Reveal the first two answers
With capacity two, threshold 0.80 flags two examples and finds one of four positives, so validation recall is 0.25. With no alerts, precision has a zero denominator and the script reports it as undefined; that is not evidence of perfect performance.
Continue with the classification metrics guide and rare events and calibration.