The Challenge

Global metrics hide local failure

Headline accuracy on the full stream can move slowly while conditional accuracy on channel_online=1 degrades faster. PSI on a single continuous feature may stay below 0.1 even when the categorical mix plot clearly shows drift—binary and low-cardinality features need dedicated tests.

Automatic retrain on every PSI alarm is expensive and often wrong. At 0.1 many teams start a review; above 0.25 often warrants escalation—but the thresholds depend on traffic volume, cost of false positives, and whether you can importance-weight new rows instead of full redeploy.

Covariate drift is not always an emergency. It is a demand to inspect segments before you schedule a full redeploy or freeze releases.

The Solution

Measure the channel, then the error

Run a fixed cadence: daily online share, weekly KS between reference and production windows, accuracy and calibration by channel and amount decile. Choose among retrain, importance weighting, or explicit channel features based on operational cost—not on whichever test fired first.

Implementation

Notebook 02 walks through mix plots, PSI/KS tables, and segment accuracy. Install data_drift with pip install -e ".[dev]", then run the cells top-to-bottom.

def accuracy_by_channel(df, y_true="label", y_pred="pred"):
    return (
        df.groupby("channel_online")
        .apply(lambda g: (g[y_pred] == g[y_true]).mean())
        .rename("accuracy")
    )

def windowed_psi(ref, cur, feature, n_bins=10):
    # reference = days 0-29, current = trailing 7 days
    return population_stability_index(ref[feature], cur[feature], n_bins)

Notebook: 02_covariate_drift.ipynb

What this run shows

Channel mix drifts upward over the synthetic horizon; segment metrics diverge before the global average moves. That ordering is what you want alarms to respect.