Modalities and governance

Layers and clocks

The Problem

LedgerRoute’s tabular head can retrain when fresh labels arrive. A text embedding served by a vendor API cannot be “retrained” inside your account when the provider ships a new encoder. Generative stacks add prompt versions, tool schemas, RAG corpora, and eval sets—each drifts on its own clock.

“Continuous learning” on the slide often means continuous governance: pin versions, measure surrogates, and decide when humans—not an automated pipeline—own the next change.

Tabular

Retrain / reweight on label join

Embeddings

Pin model ID; refresh index

Generative

Prompt hash + eval gates

Where retrain is not the lever

Embeddings, prompts, and RAG

The Challenge

Pick the lever that actually moves

Foundation weight updates are rare and deliberate. Embedding endpoints change with opaque model IDs. Prompts change every deploy. Corpus staleness is a product decision, not only an ML metric. Fraud and abuse adapt faster than monthly retrains—response time is organizational, not FLOPs.

Where teams over-promise

Weekly full retrain fantasy: Not available for vendor embeddings or large LLMs.

Single drift score: Collapses incomparable layers into one green/red light.

No pause criteria: Prediction drift pages people with no authority to stop traffic.

Raw production storage: Compliance blocks the obvious reference window for unstructured data.

Our approach

Governance loop

The Solution

Governance loop without fantasy retrain

Monitor inputs and outputs per layer. Schedule evaluations on pinned eval sets. Define shadow mode, rules-engine fallback, and human gates when prediction drift fires before labels return.

Layered maintenance model

Traffic

→

Layer monitors

→

Eval gates

→

Human sign-off

Tabular head

Retrain or reweight when labels confirm outcome drift.

Embeddings

Reference batch cosine distance; pin provider model ID.

Prompts / RAG

Hash prompts; version corpus; block deploy on eval regression.

Adversarial drift

Faster human+rules loop than monthly retrain.

What changes in operations

Pin versions

Embeddings + prompts + corpus

Surrogate metrics

When labels lag

Pause criteria

Documented before incident

Code for embedding surrogate

Distance without weight updates

Implementation

Notebook 06 scores a fixed reference batch before and after a synthetic provider swap—mean cosine distance is the surrogate we chart (no GPU LLM training in-repo).

embedding_surrogate.py

def embedding_drift(ref_vecs, prod_vecs):
    # L2-normalized rows; same expense_ids in both batches
    sim = (ref_vecs * prod_vecs).sum(axis=1)
    return 1.0 - sim.mean()  # mean cosine distance

Key points

Provider swap: Simulated shift; distance jumps when model ID changes.
Not a retrain signal: Triggers index refresh + eval gate, not weight update.
Separate monitors: Weights, embedding API, prompt hash, RAG age—independent series.

Notebook: 06_llm_embedding_surrogate.ipynb

Notebook: 05_prediction_and_delayed_labels.ipynb

Drift taxonomy (docs/taxonomy.md)

Drift FAQ — answered (docs/open_questions.md)

Summary

Close the saga

Closing the saga

Match the maintenance action to the layer. Full model retrain is one tool among many; governance is choosing which tool is allowed this week.

Embedding distance after provider swap — Mean cosine distance after a synthetic embedding provider swap—surrogate drift without retraining weights.

Per-layer monitors

Do not collapse tabular, embedding, and generative drift into one KPI.

Eval gates before traffic

Pinned sets for prompts and retrieval, not only offline accuracy months ago.

Humans own pause

Automatic rollback only where harm is bounded; else shadow + sign-off.

Questions answered

Separate monitors for LLM weights, embeddings, prompts, RAG? Yes—four series minimum. A green tabular PSI does not imply a safe prompt deploy.

When to pause: shadow, rollback, or human gate? Shadow when harm is unknown; automatic rollback only for bounded blast radius (e.g., internal copilot); human gate for customer-facing payouts and compliance queues.

Adversarial drift faster than monthly retrain? Minimum response is rules + analyst loop at hours/days; retrain is batch on labeled fraud weeks, not the first lever.

Unstructured data without raw production storage? Use hashed reference batches, embedding surrogates, and periodic labeled audits—same role as PSI for tabular.

Prediction drift without labels? Page for investigation at defined quantile shift; auto-pause only if paired with a safety metric (error budget, manual spot-check queue).

Production insights

Embedding pin

Click to reveal

Treat provider model ID like a dependency lockfile; CI fails on silent bump.

Prompt hash

Click to reveal

Deploy artifact includes prompt_hash; diff prompts like code.

RAG age

Click to reveal

Alert on corpus freshness for regulated answers, not only on click-through.

Saga takeaway

Click to reveal

Read chapters 1–4 for tabular clocks; use this chapter when retrain is not on the table.

Ready for your data saga?

We can map your label latency, segment dashboards, and governance gates to the same layered playbook—starting from the traffic you already log.

Request Free Assessment Back to saga overview