The Challenge

False alarms at scale

Monitoring dozens of features every hour multiplies false positives unless you control multiplicity. Seasonal class-balance fluctuation can look like drift until window length matches the business cycle (payroll weeks, quarter close, holiday spend).

In this seed a CUSUM on normalized daily channel mix fires mid-stream; the noise-only stream does not cross the same threshold. That separation is what you want from detector design—not identical alerts on every wiggle.

The Solution

Name the mechanism, then pick the response

Gradual covariate drift: scheduled evaluation, segment metrics, planned retrain windows. Abrupt concept or policy shift: rollback, feature flags, human gate. Noise: longer windows, stratified sampling, or seasonal baselines before paging.

Implementation

def cusum(series, threshold=5.0, drift=0.0):
    s_pos = s_neg = 0.0
    alarms = []
    for i, x in enumerate(series):
        s_pos = max(0, s_pos + x - drift)
        s_neg = max(0, s_neg - x - drift)
        if s_pos > threshold or s_neg > threshold:
            alarms.append(i)
            s_pos = s_neg = 0.0
    return alarms

Notebook: 04_shift_vs_drift_noise.ipynb

What this run shows

Side-by-side curves make the vocabulary concrete. Teams that separate runbooks respond faster than teams that retrain on noise.