Eight concepts that decide whether a prediction can be trusted.
Written for the smart non-statistician on a diligence team. Every claim we make on this site reduces to these ideas — read them once and you can pressure-test any prediction vendor, including us.
A model is calibrated when its stated probabilities match observed frequencies: of all the cases it scored at 0.70, about 70% actually resolved that way. Calibration is what turns a score into a price — an uncalibrated probability cannot be underwritten, reserved against, or allocated on, because it does not correspond to anything measurable.
Area Under the ROC Curve measures discrimination: how often the model ranks an actual favorable outcome above an actual unfavorable one. 0.5 is a coin flip; 1.0 is perfect ordering. Our public promotion gate requires AUC ≥ 0.70 on temporally held-out cases. AUC and calibration are different properties — a model needs both, which is why the gate reviews both.
Instead of randomly splitting data into train and test sets, split by time: train on earlier filings, test on later outcomes the model never saw. Random splits let a model be flattered by memorizing period-specific patterns; a temporal split evaluates it the way production will — against cases decided after everything it learned.
The background frequency of an outcome for a jurisdiction and case type. Base rates differ enormously across venues, which is why pooled national statistics mislead: the national average settlement rate is not the settlement rate in your district. Jurisdiction-specific models exist to anchor every prediction to the right base rate first.
Training-data leakage is when information that would not be available at prediction time sneaks into training — a feature that encodes the outcome, a record duplicated across train and test. Symptoms include implausibly perfect metrics. Suspiciously perfect models in our registry are held out of production for exactly this reason, and leakage findings are treated as governance events, not embarrassments to hide.
The chain of custody from a model back to real source records: which system a record came from, its source identifier, when it was ingested, how it was deduplicated. Provenance is what makes an audit possible — and its absence is disqualifying: models whose training data cannot be traced do not hold production status here.
A model overfits when it performs brilliantly on data it trained on and poorly on data it has not seen — it memorized specifics rather than learning structure. Defenses are evaluation design (temporal holdout), minimum training-data thresholds, and gates that simply refuse to promote models that cannot demonstrate generalization.
A statistical model trained to estimate the probability of defined outcomes from structured features. Deterministic, versionable, auditable, and evaluable against real frequencies — the properties underwriting requires. Contrast with generative models, which are trained to produce plausible sequences and are the right tool for language tasks, not for calibrated frequencies.