vLLM and the new default shape of LLM serving
If you are still serving with naive static batching, the gap is not marginal — paged KV-cache and continuous batching change the throughput-per-GPU math, and most other stacks have copied the idea.
Exploring the open frontier
Come learn the frontier, and build it yourself.
A learning home for working ML and AI-agent engineers who want to go deep. We run the paper, build it from scratch in runnable code, and try to explain it clearly — so the real ideas are there to understand, lean on, and build with.
New here? The welcome guide shows you around and maps out where to start. →
Essays
What a 4B Model Can Actually Do: Field Notes from 155 Experiments
Across 155 small-model experiments centered on Qwen 3.5 4B, the same thing kept working: give the model something executable it can check against the evidence it has, and it punches far above its benchmark weight. Here is the field guide — the levers that worked, how I know they're real, and the frontier they opened up.
Signals — the pulse
All signals →FlashAttention-3: async, low-precision, Hopper-native
The headline is hardware-specific: FA3 is a Hopper story (async copy/MMA overlap, FP8 paths). The portable lesson from the FA line is still the one that matters — attention is bandwidth-bound, and the win is in HBM traffic, not FLOPs.
Mamba and the selective-state-space line
Worth understanding even if you ship transformers: SSMs change the asymptotics (linear in sequence length, constant state at inference) and the failure modes. The interesting deployments are hybrids, not pure-SSM.
SGLang and RadixAttention for prefix reuse
If your workload has heavy shared prefixes — system prompts, few-shot exemplars, agent scaffolds — automatic prefix caching is close to free latency. This is where serving for agents diverges from serving for chat.
DeepSeek-R1: RL-trained reasoning with open weights
The reproducible part is the method, not a leaderboard cell: group-relative RL on verifiable rewards, with open weights to probe. It is the cleanest public artifact for understanding the reasoning-model training loop.
The modded-nanogpt speedrun and the Muon optimizer
A rare fully-public optimization target with a reproducible harness — exactly the kind of artifact we like. The Muon optimizer it popularized is the most interesting practical idea to come out of it.
Explainers — teardowns
All explainers →Reading a Model Release Like an Engineer: Weights, Licenses, System Cards, and Evals
The headline benchmark is the least durable thing in a model release. Here is how to read access, licenses, cards, eval protocols, and serving facts before you commit engineering to a number you cannot reproduce.
Post-Training Quantization in Practice: GPTQ, AWQ, and FP8
Post-training quantization is the cheapest inference lever and the easiest to pull wrong. The right method is set by your serving regime — bandwidth-bound decode wants weight-only INT4, compute-bound prefill wants FP8 — and the win is real only if a fast kernel accelerates your exact config.
GRPO, Demystified: Group-Relative Policy Optimization for Reasoning Models
GRPO swaps PPO's learned critic for a Monte-Carlo baseline — the mean reward over a group of sampled completions — trading rollout compute and per-token credit assignment for a simpler, more stable RL loop on verifiable-reward tasks.
Routing Is the Hard Part: A Practitioner's Guide to Mixture-of-Experts
MoE decouples parameter count from per-token FLOPs, but every hard problem — instability, dropped tokens, load imbalance, all-to-all traffic, a footprint set by total not active params — lives in the router. A structural tour from Switch/GShard to fine-grained and aux-loss-free designs, and the systems bill you actually pay.
Recreations — from scratch
All recreations →Recreating FlashAttention: A Tiled, IO-Aware Attention Kernel from Scratch
The minimal Triton kernel recovers FlashAttention's headline behavior — O(N) HBM traffic and bit-exact outputs against a reference — so the win we reproduce is memory bandwidth, not FLOPs; matching the CUDA kernel's absolute speedups, though, is out of reach at this scope.
Reproduced in the open — learn & trust
Browse all →Reproducing the nanoGPT Speedrun: What Actually Moves the Loss Curve
On a small config the direction reproduces — with the architecture fixed, Muon reaches the 3.28 FineWeb target in markedly fewer optimizer steps than AdamW — but the headline wall-clock record is hardware-locked to 8×H100, and our reference-node run is still pending.
Libraries — the toolshed
All libraries →TRL in Anger: SFT, DPO, and GRPO Without Rewriting Your Training Loop
TRL turns SFT, DPO, and GRPO into Trainer subclasses that inherit the entire Hugging Face stack — accelerate, peft, DeepSpeed. The convenience is real; the cost is that you're debugging someone else's training loop the moment your problem stops looking like the quickstart.
vLLM, Explained: PagedAttention, Continuous Batching, and the Serving Stack
vLLM treats the KV cache like OS virtual memory — non-contiguous paged blocks — and schedules work at the token, not the request. You get high aggregate throughput; the cost is that per-request latency becomes something you tune rather than something you get for free.
Essays — the horizon
All essays →What a 4B Model Can Actually Do: Field Notes from 155 Experiments
Across 155 small-model experiments centered on Qwen 3.5 4B, the same thing kept working: give the model something executable it can check against the evidence it has, and it punches far above its benchmark weight. Here is the field guide — the levers that worked, how I know they're real, and the frontier they opened up.
The Harness Is the Product: Why Agent Evals Are the Real Moat
Swapping the frontier model rarely moves your agent's success rate as much as fixing retries and context management — and the one thing competitors can't clone is your evaluation environment. A thesis on why agent evals, not weights, are where reproducible capability accrues.
The Economics of Thinking: Test-Time Compute as a Scaling Axis
Reasoning models turned inference into a per-request dial. This is an economic read on when spending FLOPs at test time actually buys accuracy, why it only pays where answers are cheap to verify, and what variable-cost inference does to latency budgets and capacity planning.
The Checkpoint · weekly
One thoughtful issue a week. Learn something real.
Every week: the papers, reproductions, and releases worth your attention — each taught clearly, with the code and context you need to actually use it. The good stuff, distilled, so you keep getting sharper without the firehose.