How well-calibrated are these projections?
Per-model bias · Resolved-sample counts · Known corrections
When a model says "65% chance," the actual win rate should be 65%. Below is the honest version of how close each model gets: per-model status, the size of any known miscalibration, and the correction already applied. Bin charts land as each sport's resolve-worker exposes a public endpoint.
Per-model status
CBB · Win probability
Instrumented · publishing chart soonPAV-isotonic calibrator on top of the v2 model, refit weekly. Calibrator's weak point is the 0.10–0.20 predicted-probability bin where training data is thin.
NBA · Moneyline & totals
Instrumented · publishing chart soonSeason-complete (2026). The audit found the moneyline overconfident at the extremes: where the model said 90%+, those bets hit closer to two-thirds. We replaced a broken isotonic table with a smooth Platt curve mid-season, and a shrink-to-market temperature fix is built and validated for next season. Game totals were the model's one repeatable edge.
MLB · Moneyline & totals
Instrumented · publishing chart soonMoneyline (especially NO-side) is the only validated MLB edge; the totals raw edge tested as a monotone anti-signal and was pulled. Brier reflects all resolved MLB bets, totals included.
Weather · Temperature bands
Instrumented · publishing chart soonEMOS v1 is mildly overconfident above 30% predicted probability. The live trading layer caps confidence at 0.40 in response. Public charts will show predicted-vs-actual band rates per city per band.
NBA · Totals
Instrumented · publishing chart soonLive totals residual GBM brings out-of-sample MAE from 18.2 to 11.2 (–39%) on a 1,220-game holdout. Vegas-prior τ-blend ships on top to anchor early-game projections.
NBA · Championship & series
Instrumented · publishing chart soonDual-Elo + in-series-state playoff model. The 2026 run is complete: the Knicks won the title, and the model's pre-Finals board favored San Antonio, so it missed at the very top. The 2026-06 audit confirmed favorite overconfidence in the playoffs alongside underdog under-rating; the same shrink-to-market fix targets both.
CFB · Playoff
Awaiting season dataCommittee-proxy seeding learned on 2014–2024 CFP history. Out-of-sample seed accuracy + bracket-walk Brier will land here when the 2026 season starts and we have live committee comparisons.
NFL · Season wins
LiveOff-season ratings calibrated against DraftKings 2026 win totals: 25 of 32 teams within 1.5 wins. Median team within 0.8 wins. The full per-team table is in our internal Vegas-comparison doc; cleaning a public version now.
What "calibrated" means here
Pick a model. Group every one of its predictions by predicted probability -- say in 10pp buckets. For each bucket, compute the actual rate at which the event happened. Plot those points. A perfectly calibrated model has every bucket sitting on the diagonal: where you predicted 60% you got 60%. We'll publish those plots per model here, with bin counts and rolling 30/60/90-day windows. The audit query runs Mondays at 10am PT in cbb_weekly_health_check_worker; parallel weather and NBA audits run nightly.
Why we publish the misses
The two biggest model corrections we've shipped this year -- the EMOS-v1 overconfidence cap on weather and the live NBA totals GBM replacement -- both came out of the calibration process catching the model lying. The framework is the moat. The bin charts are how we prove it's real.