How well-calibrated are these projections?
Per-model bias · Resolved-sample counts · Known corrections
When a model says "65% chance," the actual win rate should be 65%. Below is the honest version of how close each model gets: per-model status, the size of any known miscalibration, and the correction already applied.
Per-model status
CBB · Win probability
Live feedPAV-isotonic calibrator on top of the v2 model, refit weekly. Calibrator's weak point is the 0.10–0.20 predicted-probability bin where training data is thin.
NBA · Moneyline & totals
Live feedSeason-complete (2026). The audit found the moneyline overconfident at the extremes: where the model said 90%+, those bets hit closer to two-thirds. We replaced a broken isotonic table with a smooth Platt curve mid-season, and a shrink-to-market temperature fix is built and validated for next season. Game totals were the model's one repeatable edge.
MLB · Moneyline & totals
Live feedMoneyline (especially NO-side) is the only validated MLB edge; the totals raw edge tested as a monotone anti-signal and was pulled. Brier reflects all resolved MLB bets, totals included.
Weather · Temperature bands
Backtest documentedRetired 2026-08-28. Directional trading closed 2026-07-21 after 11 angles failed out-of-sample testing; the model is now off and its scheduled workers removed, so data capture has stopped too. The paper record did show an edge — 2,387 settled predictions at a 26.6% win rate against a 23.1% breakeven — but only 11.6% of live orders ever filled, the rest dying as "no taker" or "edge evaporated between scan and fill". The edge existed at prices nobody would trade. EMOS v1's overconfidence above 30% predicted probability stands as the historical note it now is.
NBA · Totals
Backtest documentedLive totals residual GBM brings out-of-sample MAE from 18.2 to 11.2 (–39%) on a 1,220-game holdout. Vegas-prior τ-blend ships on top to anchor early-game projections.
NBA · Championship & series
Backtest documentedDual-Elo + in-series-state playoff model. The 2026 run is complete: the Knicks won the title, and the model's pre-Finals board favored San Antonio, so it missed at the very top. The 2026-06 audit confirmed favorite overconfidence in the playoffs alongside underdog under-rating; the same shrink-to-market fix targets both.
NFL · Season wins
Backtest documentedOff-season ratings calibrated against DraftKings 2026 win totals: 25 of 32 teams within 1.5 wins. Median team within 0.8 wins. The full per-team table is in our internal Vegas-comparison doc; cleaning a public version now.
CFB & NFL · Game total range
Backtest documentedWe publish a range for game totals, not a pick. The claim is coverage: how often the final lands inside the band we drew. Held out, our 50/80/90% bands covered 50.1/81.1/89.9% of NFL games and 50.5/81.1/91.4% of CFB games — close to what they promise, with CFB running slightly wide at the 90% level. What the range does NOT claim is an edge: the market's number is sharper than ours (CRPS 7.29 vs 7.45 on NFL), and blending ours into the posted line buys essentially nothing. A band that misses one game in five is the band working, not failing.
McConnell's Calls: the graded record
The weekly NFL/CFB picks at /calls run on a track record, not a calibration curve. The four-slot slate (Lock, Dog, Total, Board) was replayed against 2021–2025 walk-forward predictions before a single live call was published: 90 weeks, 450 calls, slate-wide 54.0% win rate and −0.5% ROI (vig-neutral, versus −4.5% for picking every game straight). Those returns are measured against opening and consensus lines rather than closing ones, which flatters them; the live ledger grades at the number we actually posted.
| Slot | What it is | n | WR | ROI |
|---|---|---|---|---|
| Lock | favorite the model backs harder than the market | 90 | 75.6% | −9.8% |
| Dog | value underdog, biggest devig gap | 90 | 43.3% | +18.6% |
| Total | the number we disagree with most | 90 | 48.9% | −6.7% |
| Board | sides where we disagree most, filling to 5 | 180 | 51.1% | −2.4% |
These are 2021–2025 backtest priors, not this season's live results. The 2026 slate posted its first week; nothing has graded yet, so the live record on the Calls page currently reads zero. mcconnells_calls_grade.py replaces the backtest prior with the live record slot by slot once each slot reaches n≥25.
What "calibrated" means here
Pick a model. Group every one of its predictions by predicted probability, say in 10pp buckets. For each bucket, compute the actual rate at which the event happened. Plot those points. A perfectly calibrated model has every bucket sitting on the diagonal: where you predicted 60% you got 60%. That plot doesn't exist on this site yet for any model. What you see above is the resolved-count and single-number Brier score, which is real but coarser than a bin chart. The audit query runs Mondays at 10am PT in cbb_weekly_health_check_worker; parallel weather and NBA audits run nightly.
Why we publish the misses
The two biggest model corrections we've shipped this year (the EMOS-v1 overconfidence cap on weather and the live NBA totals GBM replacement) both came out of the calibration process catching the model lying. This page is how we show our work.