01 - Sport Intelligence Benchmark: Brier 0.5783 Across 97,000 Matches, V2 LightGBM Ensemble

Most football-prediction tools sold online lie about how good they are. Not out of dishonesty: they simply report the wrong metric. Accuracy (how many matches were guessed right) is the easiest statistic to make look good and the least useful one for judging a probabilistic system. Sport Intelligence, my portfolio's own football prediction engine, publishes Brier score instead: 0.5783 across 97,000 real matches. A modest, honest number that any tool claiming "80% accuracy" should beat without effort, if that percentage were actually calibrated.

Independence disclaimer: this is not a betting service and not a gambling recommendation. It is a technical benchmark of a multi-class probabilistic classifier, with code, a derived dataset, and methodology published for independent validation.

Why I dropped accuracy as the quality metric

In football 1X2 (home win / draw / away win) the class distribution is not uniform: in Europe, historically, home wins account for roughly 45% of results, draws 25%, away wins 30% (indicative orders of magnitude, not measured in this article). A model that always predicts "home win" reaches about 45% accuracy without having learned anything. A second model that correctly estimates 40/30/30 probabilities when the real outcome is genuinely balanced can score lower accuracy while being far better calibrated.

Brier score fixes this because it penalizes the distance between the stated probability and the observed outcome across all three classes at once, not just the predicted "winning" class. A model that states 90% confidence on a draw that never happens gets penalized heavily, even if it happened to guess the winning class correctly elsewhere. It is the standard metric in probabilistic forecasting literature (Brier, 1950), and it is what I use internally for production drift monitoring.

The V2 ensemble in short: two stages, 25 features, isotonic calibration

The production pipeline (app/sport_intelligence/training.py) combines 25 features per match: Elo probabilities, Dixon-Coles probabilities, market-implied probabilities, rolling form, days of rest, head-to-head history, and starting-lineup rating. A LightGBM meta-learner (with an automatic fallback to Logistic Regression when LightGBM is not available in the environment) combines those features into raw probabilities, which then pass through three independent Isotonic Regression calibrators, one per class (home, draw, away), correcting the systematic over/under-confidence typical of tree-based models.

The trade-off here is deliberate: a pure linear model would be more interpretable but would miss non-linear interactions between features (e.g. "recent form matters more when head-to-head history is thin"); a deep ensemble without calibration would produce systematically overconfident probabilities. LightGBM plus isotonic calibration is, on my 97,000-match sample, the point that gave the best trade-off between predictive power and honestly calibrated probabilities.

The number: 0.5783, what it shows and what it does not

A Brier score of 0.5783 (0.0-2.0 scale, where 0.0 is a perfect prediction) only means something against a reference point: a model that always predicted uniform 33/33/33 probabilities would score around 0.667. The gap between 0.667 and 0.5783 is the model's real, but modest, edge over pure chance, consistent with the well-documented difficulty of 1X2 prediction in football.

What this number does NOT show: it does not show the model "beating" the betting market (market-implied probabilities are themselves one of the 25 input features, so the model already starts informed by the market); it does not show performance stability over time (a separate drift monitor handles that, based on ADWIN plus Page-Hinkley over Brier residuals); and it does not show that the figure is bit-for-bit reproducible by anyone who clones the public repo, for a licensing reason explained below.

The stated trade-off: why I report an in-sample metric

The production pipeline computes the Brier score by fitting and evaluating on the same data (train_full_pipeline), not on a separate held-out split. That is a methodological choice with an explicit cost: the in-sample number is an optimistic estimate of generalization capability. For a 97,000-row dataset with a regularized model and isotonic calibration on the same fold, the gap to true out-of-sample performance tends to be modest, but it is not zero. A stricter walk-forward backtest exists internally (backtesting.py) and tracks the same metric over future time windows not seen at training time.

The public repo reflects this trade-off honestly: run_benchmark.py runs in-sample by default (to mirror the production methodology exactly) but exposes a --holdout flag that performs a stratified 75/25 split and evaluates strictly out-of-sample, so anyone cloning the repo sees both numbers instead of only the more favorable one.

The honest limits of this benchmark

The public repo does not ship the real 97,000 matches. The commercial-use license of the historical football-data.co.uk results is explicitly marked as "to be verified" in the project's own ingestion code, and another live-odds source integrated elsewhere in the pipeline explicitly forbids any redistribution, even in aggregated form. Rather than betting on an unverified reading of "derived is safe," the public repo ships an entirely synthetic sample dataset, procedurally generated under the same 25-feature schema. Anyone running run_benchmark.py on the synthetic sample gets a different absolute number (typically much lower, because a small synthetic sample is easier to fit tightly in-sample): the methodology is identical, the magnitude is not. This is documented in full in DATA_PROVENANCE.md in the repo, along with a "bring your own data" path for anyone who wants to repeat the benchmark on their own real data.

Who this kind of benchmark is for

If you are building a probabilistic prediction model over rare or imbalanced events (football, but the same applies to churn, fraud, or any multi-class classification with non-equiprobable classes), the practical advice is: always publish the Brier score, or an equivalent calibration metric, next to accuracy; state explicitly whether the number is in-sample or held-out; and compare it against a uniform baseline before claiming any "improvement." A model that beats pure chance by about 13% (0.0887 out of 0.667) is already a defensible result; one that claims "85% accuracy" with no context on the class distribution, almost never is.

In the "Original Research" series, next: 02 - GEO Retrofit Benchmark.