1. Frame the decision
A fictional subscription company wants to offer support to customers at risk of cancelling. Every Monday morning, an operations team can contact at most 500 customers. A useful first target is whether an active customer cancels during the following 30 days. The prediction must use information available at the Monday scoring time; a cancellation notice received afterward is an outcome, not an input.
Start with a business baseline: the current selection process and its contact capacity. Offline recall among the 500 highest-ranked eligible customers can measure how many observed cancellations are found, but finding likely cancellations does not by itself establish that contacting them prevents cancellation. Estimate intervention impact separately with an appropriate online comparison.
Write down the prediction time, label window, eligible population, action, capacity limit, and one possible unintended harm from contacting a customer.
2. Build a time-aware dataset
Construct one row per eligible customer and scoring date. Candidate features include activity and support contacts known at that time. Keep event time and availability time where late-arriving data matters. For a 30-day outcome, wait for the full window and any expected reporting delay before treating a customer as a confirmed non-canceller. Do not include a post-scoring cancellation event or a current account status that incorporates future changes.
Train on earlier scoring dates, tune on later dates, and reserve a recent period for final testing. Define whether the intended production population includes existing customers, new customers, or both; use an additional entity-held-out assessment if new-customer generalization matters. Fit transformations using training data only. Track the snapshot, label definition, feature code, and split boundaries.
A customer appears in several weekly snapshots. When is repeated representation appropriate, and what additional split would help assess performance for entirely new customers?
3. Compare candidates at the operating point
On a fictional validation cohort of 5,000 eligible customers, 400 cancel within 30 days. The existing process selects 500 and identifies 80 of those 400. A candidate model also selects 500 and identifies 120. The baseline recall at this capacity is 80/400 = 20%; the candidate recall is 120/400 = 30%. Precision among the selected customers is 16% and 24%, respectively. These numbers describe a 10-percentage-point increase in recall in this example, not proof of intervention benefit.
Compare both methods on the same cohort and definition of cancellation. Review performance across relevant customer groups, uncertainty, data quality, and operational constraints. A higher overall result does not override a failed mandatory slice or privacy requirement. Freeze the selection method and threshold or top-k policy before using the final test period.
If the candidate selects 500 and 120 cancel, how many selected customers do not cancel in the observed window? Answer: 380. Explain why that count does not tell you how many offers actually changed an outcome.
4. Deliver a reversible release
A scheduled batch workflow is a reasonable starting point because scoring happens weekly. Validate inputs, generate scores, select eligible customers, and write a versioned output before the operations deadline. Record model and feature versions, snapshot, counts, run status, and the decision policy. If a quality or freshness gate fails, define whether the team uses the previous approved process rather than silently delivering incomplete scores.
Before broad use, test the batch output and downstream integration. A limited rollout can reveal operational failures; a randomized comparison with a stable assignment unit and suitable guardrails is needed to estimate whether the intervention improves outcomes. Keep an approved rollback path and avoid exposing sensitive features in the contact list.
5. Monitor signals and outcomes
Immediately monitor job completion, feature freshness, missing values, score distribution, selected count, and downstream delivery. Compare serving features with a meaningful reference and investigate unexpected shifts. Since the label window is 30 days plus reporting delay, predictive quality arrives later; an input shift alone does not prove that recall has fallen.
When labels mature, evaluate the fixed selection policy on a representative cohort and relevant slices. Be alert to selective labels and to feedback from the intervention itself. Define a retraining trigger separately from promotion: a new candidate still needs data validation, evaluation, and release gates.
6. Review your architecture
- Which feature values were available at scoring time, and how is that established?
- Does the test reflect the intended customer population and label delay?
- Can the team distinguish model ranking quality from the impact of contacting customers?
- What happens if the batch misses its deadline or fails a quality gate?
- Which versioned artifacts and metrics support investigation and rollback?
Continue with pipeline gates and lineage, online experiments, and drift and monitoring.