Silent scores

The model file did not change

The Problem

LedgerRoute routes corporate expenses through a classifier that decides whether finance must review a line item. The service is healthy by the usual SRE checklist: latency within SLO, error rate near zero, and the sklearn artifact on disk has not changed checksum in weeks.

That stability is the trap. Production is not the checkpoint file; it is the joint distribution of features and labels on live traffic. When marketing shifts channel mix or procurement changes vendor categories, the frozen model can keep emitting scores while decisions quietly worsen. Reviewers see the same UI; only the error pattern moves.

In our synthetic LedgerRoute run we train a logistic regression on the first thirty days of a stable stream, then score one hundred twenty days of gradual covariate drift without retraining. The artifact bytes never change.

0.74

Minimum rolling accuracy (window 800)

0.86

Endpoint rolling accuracy (same run)

30 days

Reference training window

What we log per scored expense

prediction_row.jsonl

[
  {
    "expense_id": "exp_88421",
    "model_version": "lr_v3.2.0",
    "feature_schema": "ledger_route_v7",
    "score": 0.81,
    "features_hash": "a9f2…",
    "scored_at": "2025-03-14T09:12:04Z"
  },
  {
    "expense_id": "exp_88421",
    "label": "needs_review",
    "label_policy": "finance_policy_2025q1",
    "labeled_at": "2025-03-18T11:40:00Z"
  }
]

Without that join key you cannot reconstruct rolling accuracy when labels arrive four days later. Weekly dashboards built only on launch-month holdouts will still show ~0.78 accuracy on the training slice while the online tail is far worse.

Where monitoring breaks

Three clocks that rarely align

The Challenge

Three clocks that rarely align

Feature monitors can fire within hours when a channel mix changes. Score distributions can move when an upstream embedding provider bumps a model ID. Ground truth for whether a review was correct may arrive only after the expense closes in ERP.

Teams often instrument only the slowest clock. A Monday accuracy chart that averages January with March hides a February collapse on the online-heavy segment. The same structural gap appears on credit portfolios after macro shocks and in AML typologies that adapt faster than monthly retrains: different domains, identical monitoring mistake.

Monitoring split today (typical)

Live scores

→

Weekly accuracy
holdout only

→

Delayed labels
not joined

Why the green dashboard lies

Checkpoint worship: If the model file hash is stable, on-call assumes the model is stable.

Single-number accuracy: A global mean dominated by early months masks loss on new traffic.

Feature tests without outcomes: PSI alarms without a linked error rate produce retrain churn.

Outcome tests without features: When labels finally arrive, you cannot tell whether inputs or semantics moved.

The operational question is not “did we retrain this quarter?” It is which layer moved first: inputs, scores, or labels—and whether the magnitude justifies paging someone before the next label batch lands.

Our approach

A monitoring stack, not one chart

The Solution

Treat monitoring as a stack, not a single chart

We treat LedgerRoute like any production scorer: append-only logs for features, scores, and delayed labels with shared keys; rolling accuracy and calibration computed on the join; feature-level PSI and KS only as triage, not as automatic retrain triggers.

The goal is not to automate a redeploy on every alarm. The goal is to give humans a time-ordered narrative: “channel mix moved on Tuesday, score distribution moved Wednesday, rolling accuracy crossed the internal bar Friday when labels caught up.”

Target monitoring stack

Expense events

→

Feature + score log

→

Layered monitors

→

Human triage

Input layer

PSI/KS on high-signal features; segment by channel and region.

Score layer

Distribution of predicted risk before labels return.

Outcome layer

Rolling accuracy and ECE once labels join.

Versioning

Model artifact, feature schema, and label policy stored per row.

What changes in operations

Hours

Feature alarms (when instrumented)

Days

Authoritative outcome metrics

One join key

Required to reconcile layers

Code that makes it observable

Join scores to delayed labels

Implementation

Install the playground (pip install -e ".[dev]" from data_drift) and open the notebooks below. The cells include plots, segment tables, and verified metrics; the snippet here is the join pattern every rolling outcome metric needs.

rolling_accuracy.py

def rolling_accuracy(scored_rows, labels, window=800):
    joined = scored_rows.merge(labels, on="expense_id", how="inner")
    joined = joined.sort_values("scored_at")
    correct = (joined["pred_class"] == joined["label"]).astype(int)
    return correct.rolling(window, min_periods=50).mean()

log_and_join.py

def log_prediction(expense_id, model_version, schema, score, ts):
    store.append({
        "expense_id": expense_id,
        "model_version": model_version,
        "feature_schema": schema,
        "score": score,
        "scored_at": ts,
    })

def on_label(expense_id, label, policy, ts):
    row = store.get(expense_id)
    update_outcome_metrics(row, label, policy, ts)

Key points

Frozen weights: Same classifier throughout the drift stream; loss is environmental.
Window size: 800 rows balances noise and lag; tune to your label rate.
Do not page on PSI alone: Use prediction and outcome layers to confirm business impact.

Notebook: 02_covariate_drift.ipynb

Notebook: 01_baseline_stream.ipynb

Summary

Production readout and answers

The Production Readout

In this seed the rolling accuracy minimum is 0.74 while the endpoint rolling value is 0.86. Early stable months still weigh on the trailing window at the end of the run, which is exactly how a team can feel “fine” in stand-up while reviewers complain.

Rolling accuracy under covariate drift — Rolling accuracy with frozen weights under gradual covariate drift. The gap between minimum and endpoint values is the metric to watch.

0.74 minimum rolling accuracy

The worst short window on new traffic—not the launch holdout.

Three monitor layers

Features, scores, and delayed labels, each with its own clock.

Join key non-negotiable

Without expense_id → label, outcome drift is invisible.

Questions answered

Is prediction drift enough to page when labels are four days late? Yes, as a triage signal: shift in score distribution plus segment-level feature PSI warrants investigation. Do not auto-retrain; confirm with a shadow threshold or a small labeled audit batch when labels are delayed.

Which segments hide inside a good global average? Any channel or region whose share is growing. Plot accuracy conditional on channel_online and on amount deciles; the online tail often breaks first.

Do you version the feature schema separately from the model? You should. Store feature_schema on every prediction row. A silent column drop in upstream ETL looks like model drift if you only version the pickle.

Production insights

Dashboard trap

Click to reveal

If your accuracy chart uses a fixed holdout month, add a second series: rolling accuracy on label-joined production rows only.

Window length

Click to reveal

Match rolling windows to label arrival: too short is noise; too long hides a two-week shock.

When to retrain

Click to reveal

Retrain when outcome layer degrades and labels confirm—not when PSI alone crosses 0.1.

Replay without redeploy

Click to reveal

Replay last week’s features through the current model offline; if error reproduces, the issue is not serving infrastructure.

Continue this saga

Next chapter: When inputs move.