Stage 2 of the path. The arithmetic of a training run — what a FLOP budget buys, why scaling laws are a unit-economics argument rather than a fact about quality, what actually occupies the GPU, why your expensive silicon is idle most of the time, and what to do when a run starts misbehaving at four in the morning.
Stage 2 of five · prerequisite: The Mechanism · the path is in Intermediate to Advanced AI
Stage 1 was about what the model computes. This stage is about what it costs to make one, and almost every idea here is an economic argument wearing a research costume. That is not a criticism — it is the most useful frame available, and it is the one that makes the decisions legible.
Every number in this course is either derived in front of you or cited. Where a widely repeated figure turns out to be contested — and one of the most famous ones in the field is — the course says so rather than quietly picking a side. Two claims here are extrapolations from a single published data point, and both are labelled.
The prerequisite is real. If you cannot write the forward pass from a blank file, go back to Stage 1 — the memory accounting in Module 5 and the parallelism trade-offs in Module 7 are meaningless without it.
The single most useful fact in this stage is that the cost of training a dense transformer is approximately
The six is not magic and you should be able to derive it. Consider a single weight in a matrix multiply, processing a single token.
Two plus four is six, per parameter, per token. Multiply by all parameters and all tokens and you have the run. This is why the backward pass costs roughly twice the forward pass — a ratio worth carrying, because it is also why inference is so much cheaper per token than training, and why gradient-free methods look tempting until you count the samples they need.
Meta published both the size and the compute for Llama 3 405B, which lets us test the approximation rather than trust it. The paper states the flagship was pre-trained on 15.6T tokens with 405B parameters, using 3.8 × 1025 FLOPs.
Within a quarter of a percent, on a run that cost a nine-figure sum. The approximation ignores attention's quadratic term, all normalisation, every activation function and the embeddings, and it still lands on the published number — because at frontier scale the dense matrix multiplies genuinely are everything else rounded to zero.
Source: The Llama 3 Herd of Models (Grattafiori et al., 2024).
A FLOP count is not a plan. Three more lines turn it into one.
The only term here that is not a number you can look up is MFU — model FLOPs utilisation, the fraction of the hardware's advertised peak you actually achieve. Module 9 is about that number specifically. For now, know that large well-engineered runs land in the 30–50% range, and that assuming 100% will make you wrong by a factor of two to three in the direction that gets people fired.
NVIDIA's H100 datasheet lists BFLOAT16 Tensor Core performance as 1,979 TFLOPS for the SXM part, with an asterisk. The footnote reads: "Shown with sparsity. Specifications 1/2 lower without sparsity."
Structured sparsity requires a model whose weights have been pruned into a 2:4 pattern. Standard dense training does not qualify. The number you should plan against is about 989 TFLOPS, half the headline.
This is the most commonly misquoted specification in the field, it appears in vendor decks and capacity plans, and it makes every downstream estimate wrong by exactly 2×. When someone quotes a peak-FLOPS figure, the useful question is not whether it is right but whether it includes sparsity.
Chinchilla is usually taught as a fact about model quality. It is more useful taught as a fact about budgets, because that is the decision it informs.
Fix C, because in practice C is what is fixed — you have a cluster for a month. Since C ≈ 6ND, holding C constant makes N and D inversely proportional. You are choosing a point on a hyperbola: a large model on few tokens, or a small model on many. Both cost identically. The question is which point minimises loss.
The empirical answer from Hoffmann et al. (2022) is that the two should scale in roughly equal proportion, which for their setting put the optimum near 20 tokens per parameter. Substituting D = 20N into C = 6ND gives a formula you can use directly:
Check it on the paper's own model. Chinchilla is 70B parameters trained on 1.4T tokens, so C = 6 × 70e9 × 1.4e12 = 5.88 × 1023. Then √(5.88e23 / 120) = 7.0 × 1010 — 70B. The formula reproduces the model it came from, which is the least it should do.
The result mattered because the field had been getting this badly wrong in one direction. GPT-3 was 175B parameters on 300B tokens — 1.7 tokens per parameter, more than a factor of ten away from optimal. Chinchilla demonstrated that a 70B model trained properly beat it, using the same compute. An enormous amount of money had been spent on models that were too large and too under-trained, and the correction was worth having.
Hoffmann et al. estimated their scaling law three ways. Approaches 1 and 2 agreed on roughly equal scaling of N and D. Approach 3 — fitting a parametric loss surface, and the one whose coefficients get quoted — did not survive scrutiny.
Besiroglu, Erdil, Barnett and You (2024) reconstructed the data from the paper's own plots and found three problems. The published fit fits that data poorly. Its confidence intervals are implausibly narrow — narrow enough to require orders of magnitude more experiments than were run. And the scaling policy it implies is inconsistent with the paper's own other two approaches and with the 20-tokens-per-parameter rule the same paper recommends.
The resolution is reassuring rather than scandalous. Once optimiser and rounding issues in the fitting procedure are corrected, Approach 3 becomes consistent with Approaches 1 and 2 and with the scaling policy actually used to train Chinchilla. The 20:1 headline stands; the published parametric coefficients did not.
Two lessons, and the second is the important one. First, if you are using Chinchilla's fitted coefficients for anything load-bearing, use the corrected ones. Second: the most cited empirical result in this field had a reproducible error in it for two years, in the analysis whose numbers everyone quoted. Extend the appropriate scepticism to the next chart someone shows you, including the ones in this course.
Three limits worth stating, because the 20:1 number gets applied well outside where it was measured.
Chinchilla minimises loss for a fixed training budget. That is the right objective for a research lab establishing a scaling result and the wrong one for anybody shipping a product.
If you serve a model a trillion times, training is a one-off capital expense and inference is an operating cost that never stops. And — this is the connection to Stage 5 — inference cost during generation scales with N, because producing one token requires reading every parameter from memory. A model twice the size costs roughly twice as much per token, permanently.
So the correct move for a model you intend to serve is to go smaller than Chinchilla-optimal and train much longer. You accept a worse loss-per-training-FLOP and buy a permanently cheaper serving cost at a given quality level. The exchange rate is excellent whenever the deployment is large, and it gets better the more you serve.
Llama 3 8B was trained on the same ~15T-token corpus as its larger siblings. That is roughly 1,875 tokens per parameter — about 94× Chinchilla's ratio. Nobody made an error. An 8B model trained to 15T tokens is a deliberately different product from a 750B model trained to 160B tokens at the same compute: it is worse per training FLOP and vastly cheaper to serve, and the second property is the one that pays for a consumer deployment.
Say it in one line for a boardroom: the size of a served model is an inference-cost decision wearing a research-result costume.
"Everyone over-trains now" is the version of this that circulates, and the flagship model of the family just cited is a counterexample.
Meta's scaling-law extrapolation for their 3.8×1025 FLOP budget predicted an optimum of 402B parameters on 16.55T tokens. They trained 405B on 15.6T — essentially on their own predicted optimum, and they say so, noting that the IsoFLOPs curves flatten near the minimum at large budgets so the exact choice was not very sensitive.
So the honest statement is narrower than the slogan. Models intended for high-volume serving are heavily over-trained; frontier flagship models often are not, because a flagship's job is to establish a capability ceiling, and the serving economics are a different product's problem. Which regime you are in changes the answer completely, and "over-train, always" is advice that would have produced a worse Llama 3 405B.
Three costs, in increasing order of how often they are forgotten.
Nobody trains on a raw web crawl. The distance between "Common Crawl" and "training data" is most of the work, and it is where the largest quality differences between labs live — which is precisely why it is the least published part of the stack. A representative pipeline:
| Stage | What it does | Why it matters |
|---|---|---|
| Extraction | HTML to text, discarding boilerplate, navigation and markup. | Bad extraction poisons everything downstream and is invisible in aggregate statistics. |
| Language ID | Classify and route or filter by language. | Determines the multilingual mix, which is a product decision made in a preprocessing script. |
| Quality filtering | Heuristics and learned classifiers for "document-like" text. | Removes spam and machine-generated filler. Also the stage most likely to encode someone's unexamined taste. |
| Deduplication | Exact and fuzzy (MinHash-style) removal of repeated documents. | The highest-leverage step. See below. |
| Decontamination | Remove documents overlapping evaluation sets. | Without it your benchmark numbers are partly memorisation, and you will not know by how much. |
| Mixing | Weight sources: web, code, books, maths, curated. | Tunable, consequential, and almost entirely undocumented across the industry. |
The web is enormously redundant. The same article appears on dozens of aggregators, boilerplate licence text appears on millions of pages, and popular passages recur constantly. Left in, that redundancy does three bad things at once: it wastes compute re-learning things already learned, it increases verbatim memorisation of exactly the passages most likely to be sensitive or copyrighted, and it silently distorts the data distribution towards whatever happens to be most duplicated.
The consistent finding across the deduplication literature is that removing duplicates lets you reach a given loss with fewer tokens and reduces memorised output. It is the rare intervention that improves quality, cost and legal exposure simultaneously, which is why it is universal.
"Data quality matters more than data quantity" is repeated constantly and is too vague to be either true or false. What is well supported is narrower: deduplication and removing low-quality documents reliably help, and aggressive filtering can remove useful diversity and hurt. Where the optimum sits depends on the corpus and the target, and the labs that know have not published it.
If someone tells you their data pipeline is the reason their model is better, they may well be right, and you have no way to check. Treat data-quality claims as unfalsifiable-in-practice unless the pipeline is open.
Stage 1 covered what tokenization does. What matters here is that choosing it is irreversible: changing the tokenizer invalidates every trained weight, because the embedding table is indexed by token ID. It is fixed before the first step and lives for the life of the model.
The main lever is vocabulary size, and it is a genuine trade:
Fewer tokens per document, so a fixed context holds more text and each training step covers more material. Better compression for non-English scripts, which is a fairness and cost issue as much as a technical one.
Costs: a bigger embedding and unembedding matrix, and a more expensive final softmax over V.
Cheaper embeddings and softmax, and more training signal per token type, so rare tokens are seen more often.
Costs: more tokens per document, so more compute and context for the same text, and worse compression for anything unlike the fitting corpus.
The industry moved decisively larger — GPT-2 used about 50K, Llama 3 uses 128,256 — because at modern model sizes the embedding matrix is a small fraction of parameters (you computed this in Stage 1: 3% for Llama 3 70B, 31% for GPT-2 small) while the per-document token saving applies to every training step and every inference call. When the embedding stops being expensive, buying compression with vocabulary is nearly free.
People estimate training memory as "the model in BF16," which is off by a factor of eight. The standard mixed-precision Adam setup, as laid out in the ZeRO paper (Rajbhandari et al., 2019), costs:
| What | Precision | Bytes per parameter |
|---|---|---|
| Weights, used for forward and backward | BF16 | 2 |
| Gradients | BF16 | 2 |
| Master copy of the weights | FP32 | 4 |
| Adam first moment (momentum) | FP32 | 4 |
| Adam second moment (variance) | FP32 | 4 |
| Total model states | 16 |
Sixteen bytes per parameter, before a single activation is stored. A 7B model needs about 112 GB of model states — more than fits on an 80 GB H100, for a model whose weights are 14 GB. The ZeRO paper's worked example uses 7.5B parameters and 120 GB for exactly this reason.
It looks like waste and it is the thing that makes mixed precision work at all.
Late in training, a weight of magnitude around 1 receives an update of magnitude around
10−7. BF16 has roughly 8 bits of mantissa, so the smallest change it can represent
near 1 is about 2−8 ≈ 0.004. The update is far below that threshold, so
w + update == w exactly. The addition is discarded, silently, every step.
Keeping an FP32 master copy — roughly 24 bits of mantissa — means the updates accumulate, and the BF16 copy used for the fast matrix multiplies is cast fresh from it each step. You get tensor-core throughput on the arithmetic and full precision on the accumulation.
The failure mode without it is nasty because it is not a crash. Training simply stops improving, in a way that looks like a learning-rate schedule problem, at whatever point the updates fall below representable resolution.
Model states are only half the bill. The backward pass needs the forward pass's intermediate activations, so they must be kept. Activation memory scales with batch size, sequence length, model dimension and depth — and at long context it is dominated by the attention terms you saw in the Stage 1 ledger.
Gradient checkpointing (also called activation recomputation) is the standard answer: keep only a subset of activations, and recompute the rest during the backward pass. The usual arrangement stores one checkpoint per layer, which turns activation memory from linear in depth into roughly the cost of a single layer's activations plus the checkpoints, at the price of one extra forward pass — about 33% more compute.
That is the trade to have at your fingertips: roughly a third more compute buys you a large multiple in activation memory, which usually converts directly into a larger batch, which usually converts into better hardware utilisation. It frequently makes the run faster in wall-clock terms despite doing more arithmetic, which is a good early lesson in why FLOPs are the wrong unit.
In plain data parallelism every device holds a complete copy of all 16 bytes per parameter. Across 64 devices that is 64 identical copies of the optimiser state, which is pure waste. ZeRO shards them.
| Stage | Shards | Memory per device | Communication cost |
|---|---|---|---|
| ZeRO-1 | Optimiser states (12 of the 16 bytes) | 4 + 12/n | Unchanged from plain data parallel. |
| ZeRO-2 | + gradients | 2 + 14/n | Still unchanged — reduce-scatter replaces all-reduce. |
| ZeRO-3 | + parameters | 16/n | Roughly 1.5× — parameters must be gathered before each layer's use. |
ZeRO-3 — the same idea as PyTorch's FSDP — is the one that changes what is possible, because memory per device now falls linearly with device count. The cost is that every layer's weights are gathered from across the cluster immediately before use and released after, so you are trading memory for interconnect bandwidth. On a fast fabric that is an excellent trade. On a slow one it is a disaster, and this is the single most common reason a distributed run is inexplicably slow.
An H100 SXM can do about 989 × 1012 dense BF16 FLOPs per second and can read from its HBM at 3.35 × 1012 bytes per second. Divide them:
Read that as a threshold. If a kernel performs more than about 295 floating-point operations for every byte it reads from memory, it can keep the arithmetic units busy and it is compute-bound — the good case, where you are getting what you paid for. If it performs fewer, the arithmetic units finish and wait for data: it is memory-bandwidth-bound, and buying a chip with more FLOPs changes nothing.
That ratio — FLOPs performed per byte moved — is arithmetic intensity, and plotting achievable performance against it gives the roofline model: a diagonal bandwidth-limited region, a flat compute-limited ceiling, and a ridge where they meet.
Naive attention materialises the full T×T score matrix in HBM: write the scores, read them
back for the softmax, write the result, read it again for the value multiply. In the Stage 1 ledger
you saw that tensor is (B, H, T, T) — at batch 4, 12 heads and 1,024 tokens it is
96 MiB per layer, against a 6 MiB residual stream. It is shuttled to and from memory several
times, per head, per layer, per step.
FlashAttention (Dao, Fu, Ermon, Rudra and Ré, 2022) does not reduce the FLOPs at all. It tiles the computation so the score matrix is never written to HBM: blocks are built in on-chip SRAM, consumed immediately, and discarded, with an online-softmax formulation that keeps the normalisation correct without ever holding all the scores at once. Same arithmetic, far fewer memory round trips.
The reported gains in the original paper are 15% end-to-end on BERT-large at sequence length 512, 3× on GPT-2 at 1K, and 2.4× on long-range arena at 1K–4K. Note the pattern: the benefit grows with sequence length, because that is where the discarded tensor grows quadratically while the useful work does not.
Once you hold arithmetic intensity, you could have predicted FlashAttention. The question "which part of this computation moves the most bytes per useful FLOP?" identifies the score matrix immediately, and "can we avoid materialising it?" is the obvious follow-up.
The same question predicts most of the rest of the systems literature. Kernel fusion, mixed precision, FP8, KV-cache layout, MoE routing, gradient accumulation — nearly all of it is someone asking where the bytes are going. The paper's title says IO-awareness rather than attention for a reason, and the reason is the whole lesson.
| Kind | Splits | Buys | Costs |
|---|---|---|---|
| Data | The batch. Every device holds the whole model. | Simplest, near-linear throughput scaling. | Nothing on memory — every device still needs everything. One gradient all-reduce per step, which overlaps well with the backward pass. |
| Tensor | Individual weight matrices, within a layer. | Fits a layer too large for one device; reduces per-device memory and compute together. | Communication inside every forward and backward pass, twice per block. Needs NVLink-class bandwidth; degrades badly across nodes. |
| Pipeline | Layers across devices. | Fits a very deep model with modest communication — only activations cross the boundary. | Bubbles. Devices idle waiting for the stage ahead. Micro-batching shrinks the bubble; it never removes it. |
| Sequence / context | The sequence dimension. | Contexts whose attention memory would not otherwise fit. | Attention needs all positions, so keys and values must circulate — typically a ring exchange overlapped with compute. |
Real systems compose all four. The arrangement that keeps recurring is:
Tensor parallelism inside a node. Pipeline parallelism between nodes. Data parallelism over everything. Sequence parallelism when the context demands it.
The reason is a bandwidth hierarchy, and once you see it the rule stops being arbitrary. Inside a server, GPUs talk over NVLink at hundreds of gigabytes per second. Between servers they talk over InfiniBand or Ethernet at a small fraction of that. Tensor parallelism communicates twice per transformer block, so it must live where bandwidth is cheapest. Pipeline parallelism communicates once per stage boundary and only activations, so it tolerates the slow link. Data parallelism communicates once per optimiser step and overlaps with computation, so it tolerates the slowest link of all.
Match the communication frequency to the bandwidth hierarchy. That is the whole principle, and it is enough to reconstruct the rule from first principles when you have forgotten it.
With P pipeline stages and M micro-batches, the fraction of time devices spend idle is approximately (P − 1) / (M + P − 1).
Four stages and four micro-batches: 3/7, about 43% of your cluster idle. Four stages and thirty-two micro-batches: 3/35, under 9%. Same hardware, same model, a five-fold difference in waste from one scheduling parameter.
Which is why the pipeline-heavy configurations you will read about always come with a large micro-batch count, and why raising the micro-batch count is the first thing to check when a pipelined run underperforms. It also explains why increasing pipeline depth without increasing micro-batches makes things worse, which is a mistake people make when they are chasing a memory problem and forget they have a throughput one.
This is the interview question for this stage, and the reason it works is that people who have trained something have ranked opinions while people who have only read produce a memorised list. Here is a defensible ranking, cheapest diagnostic first.
The PaLM paper reports spikes in the loss roughly 20 times during training of the 540B model, at highly irregular intervals, despite gradient clipping being enabled. Their mitigation was to restart from a checkpoint about 100 steps before the spike and skip roughly 200–500 data batches, which worked — the loss did not spike again at the same point.
Here is the part worth remembering. They did not conclude the spikes were caused by bad data. They ran the ablation: take the batches surrounding the spike, train on those same batches from a different, earlier checkpoint, and see what happens. No spike.
So the spike was not a property of the data and not a property of the parameters. It was a property of the interaction — a particular batch meeting a particular parameter state. That is an uncomfortable result, because it means "find the bad data" is often the wrong investigation, and it means these events are not fully reproducible even in principle from the data alone. Anyone who tells you loss spikes are simply a data-cleaning problem has not read this ablation.
At small scale a GPU failure is a bad day. At cluster scale it is a rate, and you engineer for it. Meta published the numbers for a 54-day snapshot of Llama 3 405B pre-training on up to 16,000 H100s:
Roughly one unexpected interruption every three hours, sustained for two months, and they still kept the cluster productive more than 90% of the time. That is not luck. It is fast checkpointing, automated detection and node draining, and restart machinery treated as core infrastructure rather than as operations overhead.
The practical consequence for anyone planning a run: checkpoint frequency is a function of your failure rate, not of your preferences. If you expect an interruption every three hours and you checkpoint every six, you are on average discarding an hour and a half of cluster time per failure — at 16,000 GPUs that is a genuinely large number, repeated hundreds of times.
The numerator is the useful arithmetic your model requires, from Module 1. The denominator is what the hardware could theoretically do. The ratio is the fraction of your very expensive silicon that is doing work you actually needed.
It is the right headline efficiency metric because it is hardware-independent in the numerator and model-independent in the denominator, so it compares across both. It was introduced in the PaLM paper (Chowdhery et al., 2022).
Model FLOPs utilisation counts only the FLOPs the model logically requires. Hardware FLOPs utilisation counts every FLOP actually executed, including recomputation from gradient checkpointing and redundant work in tensor parallelism.
HFU is always the larger number. PaLM reported 46.2% MFU and 57.8% HFU for the same run — an eleven-point gap, and it is not noise: the gap is mostly the extra forward passes bought by activation recomputation, which is real work the hardware did and not work the model needed.
Both are legitimate metrics for different questions. HFU tells you how well your kernels use the chip; MFU tells you how much you are paying per unit of useful training. If a vendor quotes a utilisation figure without saying which, assume HFU, and ask.
| Run | Reported | Note |
|---|---|---|
| PaLM 540B, 6,144 TPU v4 chips | 46.2% MFU / 57.8% HFU | The paper that introduced the metric; it also notes prior systems struggled to exceed 30%. |
| Llama 3 405B, 8,192 H100s | 43% BF16 MFU | The higher of Meta's two reported configurations. |
| Llama 3 405B, 16,384 H100s | 41% BF16 MFU | Meta attributes the drop to a smaller per-replica batch needed to hold global batch size constant. |
That last row is the most instructive line in the table. Doubling the cluster cost two points of MFU — not because the engineering got worse, but because holding the global batch size fixed while doubling data-parallel width halves the per-replica batch, and smaller batches have lower arithmetic intensity. Scaling out is not free, the inefficiency is structural, and it is predictable from Module 6.
Use these as your calibration. If you measure 15% MFU, you have a real problem worth a day of profiling. If you measure 40%, you are in the same band as published frontier runs and your time is better spent elsewhere. If you measure 85%, check your arithmetic — you have probably used the sparsity peak in the denominator.
Every formula in this tool appeared in Modules 1 through 9, and each output says which one produced it. The tool asks you to guess the compute-optimal model size before computing one, for the same reason as always: an answer you see first is one you will find reasons to have expected.
A cluster, a calendar and an efficiency assumption. That is all a training run is.
Same compute, different allocation. Move the ratio and watch training quality trade against serving cost — which is the entire argument of Module 3, as a number.
Grounded: the FLOP model (C = 6ND) is checked against Llama 3's published compute to within 0.3%. The peak-FLOPS and bandwidth figures are from vendor datasheets, using dense not sparsity numbers. The MFU calibration band comes from PaLM and Llama 3's published values. The memory accounting is the ZeRO paper's 16 bytes per parameter.
Extrapolated from a single data point: the hardware-interruption estimate scales Meta's Llama 3 figure — 419 unexpected interruptions over 54 days on 16,000 H100s — linearly in GPU-days. That assumes your failure rate matches theirs, which depends on your hardware vintage, your data centre and your software stack. Treat it as an order of magnitude, and notice that it is the only number here derived from one observation.
Rough: the serving cost model assumes decode is purely bandwidth-bound, weights are sharded across the minimum number of GPUs that fit them, and nothing else competes for memory. It ignores the KV cache entirely, which Stage 5 will show is usually what actually limits concurrency — so this figure is optimistic, and knowingly so. It is here to show the shape of the training-versus-serving trade, not to price a deployment.
Absent: data acquisition and cleaning cost, salaries, failed runs, hyperparameter sweeps, and the several smaller models you train first. In a real programme these are frequently larger than the headline compute line.
Three artefacts and one diagnosis.
Then the diagnosis, which is the actual test: your loss spikes at step 4,000 and does not recover. Name five candidate causes, ranked, and the cheapest diagnostic for each. If you have trained something you will have opinions and they will be ordered by what you have actually checked at 2 a.m. If you have only read about training, you will produce a list.
You have cleared Stage 2 when you can be handed a run that is behaving badly and have a next move — and when you can size a run from a budget without opening anything.
Not the test: having run train.py to completion. The distinguishing
capability at this stage is diagnosis, not execution.