methodology

what we run, how, and why you can check us

generative-only

Every task is scored from chat completions using the model's own chat template — never loglikelihood. This keeps scoring identical whether a model is served by vLLM or llama.cpp, and matches how models are actually used.

as-served precision

We evaluate models at the precision people actually run them (FP8, NVFP4, GGUF quants), and every published number carries a badge saying which. We do not evaluate below an integrity floor (~4.25 bits per weight). A quarterly anchor study publishes the FP8-vs-4-bit delta on a reference model so the effect of quantization is measured, not argued.

the battery

Capability: MMLU-Pro (5-shot CoT), GPQA-Diamond, MATH-500, IFEval, HumanEval+/MBPP+ (evalplus, sandboxed). Safety/bias (bias.md): BBQ, XSTest, StrongREJECT. Task versions are pinned and printed on every report. When a run would exceed its compute window, a disclosed stratified subsample is used and flagged on the affected numbers.

judging

Judge-scored tasks use a pinned local judge model, greedy decoding, published prompt SHAs, archived transcripts. Same-family judging is disclosed on the report.

reproducibility

Every report ships repro.json (model revision SHA, engine image digest, verbatim launch command and environment, harness commit, task versions, template SHA, seeds, judge SHAs, hardware) and a runnable reproduce.sh. The promise is a replayable procedure with stated tolerances — not bit-identical output.

independence

Nobody pays for placement. Nobody previews reports. Corrections are public and permanent.