01ResearchPreprint — working draft

BM-Bench

A distributional and operational benchmark for generative simulators of the GB electricity Balancing Mechanism, built on five years and 376 million rows of market data.

← Back to projects

BM-Bench asks a question most simulator papers skip: does a simulator that matches the market statistically also train agents that work on real data? The answer, measured across three simulator classes and nine trained reinforcement-learning agents, is no — and the gap between the two is the finding.

System proof

Measured results

Three simulator classes — bootstrap resampling, a first-order Markov chain, and a causal Transformer world model — evaluated on the same held-out seven months of 2025 market data, both distributionally and by the real-data performance of PPO battery-dispatch agents trained inside each.

Distributional score

World model wins 9 of 10 stylized facts with non-overlapping 3-seed bands; 3× lower price Wasserstein distance, 4× lower diurnal profile error

Sim-to-real transfer

World model: zero transfer gap (in-sim £+72/wk vs real £+99/wk). Markov: −75%. Bootstrap: collapse to loss-making

Regime stability

World model Sharpe identical (1.41 → 1.41) across both halves of the test window; Markov degrades 1.72 → 1.00

Statistical rigor

Pairwise Diebold-Mariano under Newey-West HAC variance, p ≪ 10⁻¹⁰; five-window walk-forward validation

Reproducibility

End-to-end REPRODUCE notebook from public API to headline table, plus a committed methodology audit and remediation log

BM-Bench

What it is

The Balancing Mechanism is the auction Great Britain's electricity system operator uses to balance supply and demand in every half-hour settlement period. Anyone training a reinforcement-learning agent for battery dispatch or wind bidding needs a simulator of it — and until now there was no published way to test whether such a simulator is any good.

BM-Bench adapts the LOB-Bench methodology (Nagy et al., ICML 2025), built for equity limit order books, to five years of Elexon BMRS data: twelve distributional stylized facts spanning price and volume distributions, autocorrelation structure, diurnal profiles, and market-concentration statistics, evaluated across three simulator classes with three random seeds each.

BM-Bench

The finding that matters

Distributional fidelity and operational usefulness turn out to be separable axes. A hand-designed Markov chain posts the highest in-simulator training reward — and loses 75% of it when the trained agent meets real data. A bootstrap simulator trains agents that lose money outright. A small (53K-parameter) Transformer world model trains agents whose real-world performance equals their in-simulator performance, with identical Sharpe across both halves of a seven-month held-out window.

The naive hypothesis — higher training reward in the simulator implies better real performance — is empirically false on this data. That is measured with Diebold-Mariano tests under Newey-West heteroskedasticity-and-autocorrelation-consistent variance, significant at p ≪ 10⁻¹⁰.

/376 million rows of Elexon BMRS data, 2021–2025, pulled via an open-source client
/Walk-forward validation across five rolling train/test windows
/Gate-closure-safe forecast pipeline after catching a 32% look-ahead selection bias in-house
/A falsifiable failure-mode ablation: four hypotheses, three interventions, one clean root cause

BM-Bench

Why the methodology is the point

The project's posture is that weak evidence should be hard to hide. Every headline number carries seed error bars; a pre-submission methodology audit and its remediation log are committed alongside the paper; and the repository ships a reproduction notebook that rebuilds every result from the raw public API, with a verification cell that checks the headline numbers at the end.