benchgap
Mind the Gap: Explicit Cross-Benchmark Calibrations for Estimating Missing LLM Evaluation Scores
Most LLMs are evaluated on only a few benchmarks: across 401 models and 146 benchmark versions, coverage is just 8.3%. benchgap estimates the missing scores without running any evaluations, calibrating each benchmark against same-capability benchmarks with monotone curves selected by leave-one-out cross-validation, and attaching an error and confidence level to every estimate. In a 15% holdout it fills 71% of masked cells at 4.6 pp MAE, 23% lower than a matrix-completion baseline. It also quantifies the harness tax, the disagreement between evaluation setups on the same models. Estimates are updated daily at benchgap.net.