The field ships an astonishing amount of results. Every frontier release arrives with a launch thread, a benchmark table, and a dozen thoughtful recaps before lunch — and honestly, that abundance is a gift. The hard part isn’t finding results anymore; it’s reading them well enough to know which ones you can build on. That skill — reading an ML result without fooling yourself — is completely learnable, and teaching it is the reason this publication exists.
Here’s the friendly version of our whole method: we build the thing and run it, on hardware we name, with a config we pin, and we show you where the claim held and where it bent. Not to catch anyone out, but to learn from it and to hand you something you can trust without re-deriving it yourself. Reproduction is just curiosity with a receipt.
What a benchmark number is actually telling you
A score like “80.5 on MMLU” feels like a fact about a model. It’s really a fact about a model and an entire experimental setup, and the setup does a surprising amount of the talking. Change the prompt template, the few-shot count, the answer-extraction regex, or the decoding temperature, and the same model’s MMLU number can move by more than the gap most launches are celebrating. None of that is anyone cheating — it’s just that “the benchmark” is shorthand for a dozen choices, and two papers rarely make the same dozen.
So when you read a number, the generous and useful habit is to read the setup alongside it. Was it 0-shot or 5-shot? Greedy decoding or sampled, and at what temperature? What scoring code turned a free-form generation into a right-or-wrong? When those are disclosed, the number becomes comparable and you can do real work with it. When they’re missing, the number isn’t wrong — it’s just not yet a number you can stand on, and knowing that before you spend a sprint on a technique is half the battle.
Four questions that make a result legible
Before we write strongly about a release, paper, or benchmark, we ask four friendly questions. None of them is a pass/fail gate — together they tell us how confidently we get to speak, and they’re the same questions worth asking on your own.
Can it be re-run?
Is there enough to rebuild it — code, configs, seeds, data provenance, the actual hyperparameters, not just “we used Adam”? A method described in prose is a wonderful idea, and we love ideas; it’s simply that an idea and a runnable artifact deserve different levels of confidence. The thing we celebrate most is a repo that builds and a single command that recovers the headline figure.
What’s actually released?
Open weights, open code, API-only, or a slide in a keynote — each one opens a different door. With weights you can probe, quantize, and re-eval; without them you’re reading a careful description of a system rather than the system itself. Both are worth your time; they just support different kinds of claims. [1]“Open weights” and “open source” are happily not the same thing. Training data, the data-mixing code, and the full recipe are almost never released, even by labs generous enough to publish weights. Keeping the two ideas distinct saves you a lot of confusion: what you can fine-tune is not the same as what you can reproduce from scratch.
Is the eval described well enough to compare?
Prompt format, few-shot count, decoding parameters, scoring code, contamination controls. When these are disclosed, a leaderboard cell becomes a result you can reason about and line up against others. When they’re not, it’s a promising signal that’s still waiting for its context.
Will it run where people actually work?
Tokenizer quirks, nominal versus effective context length, quantization support, whether vLLM or TGI or TensorRT-LLM picked it up on day one. A model that’s hard to serve is genuinely interesting research; knowing that up front just helps you set honest expectations about shipping it.
A friendly status legend
When we publish a reproduction, we tag it with one of six statuses. Think of these as a weather report on how far along the work is — not a verdict on anyone’s competence, just an honest signal of how much weight a given sentence can carry.
| Status | What it means | What you can lean on |
|---|---|---|
| Reproduced | We, or a credible third party using open artifacts, re-ran it and landed in the same place within a stated tolerance, on hardware we name. | A runnable config, a pinned harness, and results that match the claim’s direction. Exact parity isn’t the bar — agreement on the trend is. |
| Partly there | Some of the claim held beautifully; another part landed differently — the trend matched but the magnitude didn’t, or one sub-result carried and another didn’t. | A scoped run that shows exactly which pieces held and where the boundary is. This is where most honest reproductions live, and that’s a good thing. |
| In the lab | We’re actively running it right now; numbers are coming in and we’re labeling our own as in-progress. | The public records we’re checking against, plus our own numbers marked provisional. |
| On the bench | It’s set up and queued — we’ve read it, we want to build it, and it’s next in line. | An honest “we haven’t run this one yet, and here’s what we’ll check when we do.” |
| Didn’t reproduce (yet) | A good-faith run with a disclosed config didn’t recover the result, so we publish the whole command and invite a fix. | The full config and command, in the open, so authors or readers can show us what we held wrong. The “yet” is sincere. |
| Still being worked out | Credible people disagree, reproductions conflict, or a methodological question is genuinely open. | Links to the competing evidence and a plain statement of what’s still up in the air. |
A couple of things this legend quietly encodes. “On the bench” and “Still being worked out” are the normal state of most of the world at any given moment, including things that are almost certainly true — they’re a to-do list, not a doubt. “Partly there” is where most honest reproductions land, and that’s the happy case: our from-scratch FlashAttention recreation recovers the kernel’s memory-scaling behavior without matching the reference’s absolute speedups, because that last mile is months of kernel engineering we simply haven’t done yet. Our nanoGPT-speedrun reproduction sits in the lab while the runs finish — we report the public records and mark our own numbers in-progress, because telling you what we know and what we’re still measuring is part of the fun.
Contamination, and why fresh problems matter
Here’s one of the most useful things to internalize early: a benchmark can be perfectly fair and still flatter a model, because test sets leak into pretraining corpora. A model that has quietly seen the answers will hand them back, and from the outside that looks exactly like skill. This isn’t a conspiracy — it’s what naturally happens when public problems and public web crawls share the same internet.
The cleanest demonstration is Scale’s GSM1k, a held-out set built to mirror GSM8K’s difficulty and style without sharing any of its problems. Several model families scored meaningfully lower on the fresh GSM1k problems than on the original GSM8K — which is exactly the fingerprint of overfitting to a public benchmark. [2]A useful default: treat any benchmark older than a model’s training cutoff as contamination-suspect, and weight private or freshly-built held-out sets more heavily than the canonical public ones. It isn’t cynicism; it’s just controlling for an obvious confound. The practical takeaway for your own reading is a kind question to keep handy: when a model looks great on a famous old benchmark, ask how it does on something it couldn’t have memorized.
Pin your harness so the number holds still
Scores drift across harness versions in ways that have nothing to do with the model — a changed prompt template or extraction rule between releases is enough to move things. The fix is delightfully simple and worth adopting in your own work: pin the harness, set a seed, and commit the exact command, so future-you (and everyone else) can recover the same number. Here’s the pattern we use, and it’s just as useful in your own repo:
# Pinned, seeded, committed — the command is part of the result.pip install "lm-eval~=0.4" # pin the harness; scores drift across versions
lm_eval \ --model vllm \ --model_args "pretrained=org/model,dtype=bfloat16,seed=1234" \ --tasks gsm8k \ --num_fewshot 5 \ --batch_size auto \ --output_path "runs/gsm8k_$(git rev-parse --short HEAD).json"The ~=0.4 keeps you on a compatible harness line so a future patch doesn’t silently move your numbers; the seed makes the run repeatable; and stamping the output with the git hash means every result traces back to the exact code that produced it. Do this once and a benchmark stops being a vibe and becomes something you — or a reader — can rebuild from scratch.
Reading an API-only release
Most frontier models are API-only at launch, and that’s genuinely fine — there’s a lot you can learn without weights. You can take the claim seriously and attribute it. You can read the system card closely for what it discloses, and notice what it leaves out — training-data description, eval methodology, known limitations, safety evaluations — because the shape of the silence is informative too. And you can place the claim next to open results you’ve actually verified and see whether the story hangs together.
What you can’t yet do is confirm it from the outside: you can’t probe the model, check effective context against the nominal number, test for contamination, or quantize it and measure the drop. So an API-only release sits in “still being worked out,” and — this is the part we like best — we say plainly what would move it. “Open the weights, or publish the eval harness, and we’ll run it.” That’s a standing, sincere invitation, not a dig.
Corrections are upgrades
We will be wrong sometimes — reproductions are run by humans on finite budgets, in a field that moves faster than any review process. So we treat a correction as a feature, not an embarrassment: every one is dated, logged in a visible changelog, and appended rather than silently edited. The original claim stays legible with a strike and a note, so you can watch how our understanding improved and why. Thinking in public means showing the diffs in your own reasoning, cheerfully.
This is also why we date evergreen methodology and specific claims separately. A guide like this one is meant to last, while a result about a particular model is stamped “as of” a date, because it has a shelf life and you deserve to know it.
The bottom line
Reading ML results well is a craft, and the good news is that it’s entirely teachable: read the setup behind a number, ask how a model does on problems it couldn’t have memorized, pin your harness, and treat a reproduction as something you build rather than something you take on faith. We do this work in the open so you can learn the moves and reuse them — and so that when we hand you a number, you can act on it without re-deriving it yourself.
The best part is that it’s a two-way street. If you’ve reproduced something we got wrong, send the config; a correction with evidence is the highest-status contribution anyone can make here, and we’ll date it, log it, and thank you in the changelog. We build it and run it so you can learn from it and trust it — and teaching each other how not to fool ourselves is the whole point.