When the label rule changes
Policy shock on day 70
The Problem
On day 70 of the synthetic stream, finance enables a stricter review policy. Categories that auto-cleared now require human eyes. Input features look similar; the correct label changes.
That is concept drift: P(Y|X) moves. Retraining on pre-shock labels teaches the old world. Mixing pre- and post-policy rows in one loss without a version bit looks like noise instead of a regime change.
Where accuracy misleads
Calibration vs queue volume
The Challenge
Accuracy and calibration diverge
In this run, hold-out accuracy is slightly higher after the shock (~0.91 vs ~0.78 pre) while expected calibration error rises from about 0.008 to about 0.12. A team watching only accuracy would miss that predicted probabilities no longer mean “probability of review under current policy.”
Thresholds tuned on old score-to-outcome curves send the wrong volume to reviewers: too few after a tightening, too many after a loosening. Prior shift—class balance changes without semantic change—needs threshold tuning, not necessarily a full retrain.
What breaks in production
Our approach
Version labels like schema
The Solution
Version the label rule like a schema migration
Store label_policy with every label. Monitor calibration and queue depth by policy version. Schedule relabel or retrain on post-shock data; during transition, route borderline scores to rules or humans with an explicit flag.
Policy version field
Treat like feature_schema—required on every label.
ECE dashboards
Page on calibration before accuracy moves.
Transition week
Shadow thresholds; do not auto-apply old cuts.
Prior vs concept
Balance-only change → thresholds; semantic change → labels.
What changes in operations
Code for calibration by policy
ECE and rolling accuracy
Implementation
def expected_calibration_error(y_true, y_prob, n_bins=10):
bins = np.linspace(0, 1, n_bins + 1)
ece = 0.0
for lo, hi in zip(bins[:-1], bins[1:]):
mask = (y_prob >= lo) & (y_prob < hi)
if mask.sum() == 0:
continue
acc = y_true[mask].mean()
conf = y_prob[mask].mean()
ece += mask.mean() * abs(acc - conf)
return eceKey points
- Shift day 70: Marked in stream metadata and figure JSON sidecar.
- Rolling accuracy: Can look healthy while ECE jumps—plot both.
- Retrain data: Post-shock labels only, or weighted by policy version.
Summary
Semantics and thresholds
What this run shows
Concept drift is a semantics problem. The model may rank rows consistently while the mapping from score to action is wrong for the new policy.
Calibration broke first
ECE 0.008 → 0.12 while accuracy moved the other way.
Policy as schema
label_policy must ride with every label row.
Thresholds are not universal
Recalibrate per policy version, not per model file only.
Questions answered
Production insights
Finance releases
Click to revealPut policy launches on the same calendar as schema migrations—with rollback owners.
Reviewer queue
Click to revealQueue depth is often the first human-visible symptom of concept drift.
Relabel budget
Click to revealPlan relabel hours when policy tightens; old labels are actively harmful.
Accuracy trap
Click to revealHigher accuracy after a tightening can mean more true positives under a stricter rule—not a better score scale.
Continue this saga
Next chapter: Shift, drift, and noise.