Topic

Optimization

Optimizers, learning-rate schedules, and training dynamics.

2 checkpoints

CHECKPOINT 00102026-06-15REPRODUCTIONSIn the lab

Reproducing the nanoGPT Speedrun: What Actually Moves the Loss Curve

On a small config the direction reproduces — with the architecture fixed, Muon reaches the 3.28 FineWeb target in markedly fewer optimizer steps than AdamW — but the headline wall-clock record is hardware-locked to 8×H100, and our reference-node run is still pending.

2026-06-11SIGNALS

The modded-nanogpt speedrun and the Muon optimizer

A rare fully-public optimization target with a reproducible harness — exactly the kind of artifact we like. The Muon optimizer it popularized is the most interesting practical idea to come out of it.

pretraining optimization reproducibility