.jpg)
A silent partition failure in an upstream data source caused over a year of historical risk metrics to drop from our processing window without triggering an alert.
We resolved the root cause, hardened our ingestion pipelines, and implemented multi-tiered validation across every external data dependency.
Then we realized we had missed an entire category of edge cases.
The assumption that breaks in production
The standard mental model of ML monitoring is this: control data quality at ingestion, catch model drift at training time, and the outputs look after themselves.
For most ML applications, that is defensible. A recommendation system that drifts slightly still functions. The damage is gradual and recoverable. But when model outputs are consumed directly by a live pricing engine, the calculus changes.
Our risk scoring models produce output tables that the pricing engine reads at quote time. A score is generated, that score maps to a rating factor, and a premium is set. There is no human in the loop for most quotes. If a scoring table develops an integrity problem, pricing acts on the bad scores until someone notices.
The failure modes are not exotic:
None of these fail a structural validator. None show up in input monitoring. All of them move live prices.
Output tables are operational data, not just model artifacts
The insight that drove this work is simple: the moment a model output table is read by a production system, it has crossed the boundary from model artifact to operational data. At that point, it deserves the same monitoring discipline we apply to any other data source in the pricing pipeline.
That means three things.
Distribution drift checks. For each output table, we compute the score distribution over rolling windows and compare it against the baseline. We are not looking for training drift. We are looking for population-level shifts in the scored output that could indicate a pipeline issue, a scoring boundary problem, or a data dependency change.
The key question is not whether the model changed. It is whether the distribution of what pricing is actually reading changed.
Integrity checks. Each output table has a defined valid range per score column and a minimum coverage expectation: the percentage of entities that must receive a score on each scoring run. If coverage drops below threshold because a scoring run silently skips a cohort, the integrity check catches it before pricing consumes the table.
We also check for null creep: the slow accumulation of null scores that happens when an upstream feature disappears or when a model is called with an entity type it was not designed to handle.
Stability validation. For entities scored on consecutive days without any change to their underlying data, scores should be stable within a defined tolerance. Instability points to either a non-deterministic model artifact or a feature pipeline that is not behaving idempotently.
This is the check that surfaces stale-feature bugs: a job ran successfully, logged a clean exit, but computed scores on the wrong feature snapshot.
A concrete example
An entity is scored on Monday and Tuesday. Its underlying data has not changed. The expected output is a stable score within defined drift tolerance.
If the stability check fires, we trace back: did the feature snapshot differ between runs? Did the model version change? Did a lookup table update mid-run? The error surfaces at the output layer, before it propagates to a premium.
Without this check, the only signal is a downstream anomaly: a sudden distribution shift in written premiums, or a reviewer noticing an account is priced differently on consecutive requests. Both signals are late, noisy, and hard to root-cause.
Keeping monitoring current as models ship
A persistent operational challenge with output monitoring is keeping it synchronized with model releases. When a new scoring variant ships, the monitoring checks need to update as well. Leaving that as a manual step means monitoring always lags the model.
We automated the onboarding using a Claude Code skill that runs when a new model variant is ready for production. It reads the output schema, computes baseline distribution statistics from a validation set, configures the drift and integrity checks with sensible defaults, and generates a Notion doc with initial analysis: score distribution histograms, coverage metrics, cross-variant comparison where applicable, and a flagged list of gaps that need human review before go-live.
The result is that the first production scoring run for a new variant already has checks in place. Monitoring configuration does not fall behind the model release cadence.
Making a check cheap to add
Everything above assumes the checks exist. In practice, the effort of writing them is what determines whether coverage is comprehensive or aspirational.
Authoring a single check used to be a small project. It meant editing the dbt configuration by hand, writing the check SQL against fully-qualified production tables, encoding pass, warn, and error thresholds, assigning a category and a schedule, dry-running it through the dbt CLI, and opening a pull request. Each check was tractable on its own. In aggregate, the overhead meant checks that should have existed did not, and coverage lagged the tables that needed it.
We removed that overhead by putting an agent in front of the check configuration.
An engineer describes the check in plain language: monitor this score column for null creep, enforce a valid range on that table, alert this group on failure. The agent reads the table's most recent EDA profile (the same distribution statistics the launch onboarding uses) and calibrates thresholds against observed distributions rather than guesses. It decides whether the check should block the pipeline or log a warning, drafts the SQL, resolves the alert recipients, and runs the check against the warehouse to confirm it returns a sensible value before presenting anything. What the author reviews is a validated draft, not a blank configuration file.
The workflow is deliberately constrained. The agent proposes; it does not write to production on its own. New checks are applied to the development environment first, and promotion to production is a separate, permissioned action. Its tools are sandboxed to the repository and to a fixed set of safe commands, and every draft is validated server-side before it can be approved. Checks that block the pipeline are opened as pull requests automatically, with no local checkout and no credentials in circulation.
Where the previous section keeps a new model's checks from lagging its launch, this generalizes the same idea to every check in between: when the cost of writing a check approaches the cost of describing it, coverage stops being a question of effort and becomes a question of judgment, which is where it should be.
What the framework has caught
Since adding output-layer monitoring, we have surfaced issues that input-layer validation would have missed.
None of these would have been visible from training metrics alone.
What this does not cover
Output monitoring catches population-level signals and per-entity stability. It does not catch correctness at the individual entity level. If a model produces a plausible but wrong score for a specific entity, that requires ground truth comparison, which lives in a separate validation layer tied to outcome data.
It also does not replace periodic model calibration. Distribution stability within acceptable bounds does not mean the model is performing well on population segments that have shifted since training. You still need regular calibration runs against recent outcome data to catch silent model degradation that manifests only at the business metric level.
The monitoring boundary has moved
The immediate roadmap extends this framework to additional scoring tables across our other program lines, following the same three-tier pattern.
The broader principle: in any ML-intensive production system, the boundary between model artifacts and operational data is a monitoring boundary. Treating that point with the same rigor applied to external data ingestion, structural validation, anomaly detection, drift awareness, turns model correctness from an assumption into a measurable property.
Once a score is read by a pricing engine, it is production data. Monitor it like production data.
