Stage 3 of the path. How a document completer becomes an assistant — SFT, RLHF, and the derivation of DPO's closed form done properly rather than asserted. Then the part that gets skipped: exactly where that closed form bites, why the probability of the response you preferred can fall during training, and why reward moving from taste to verification is what actually unlocked reasoning models.
Stage 3 of five · prerequisite: The Training Stack · the path is in Intermediate to Advanced AI
Post-training is where the most confident wrong beliefs in applied AI live, and one of them — that fine-tuning teaches a model facts — costs enterprises more money than any other single misconception in the field. Module 3 is about that specifically.
The DPO derivation in Module 5 is done in full. Most explanations assert that the reward model "cancels out." It does, for a specific reason, in a specific step, and you should be able to point at that step. Module 6 then does the thing most treatments skip entirely: what the closed form costs you, with a mechanism you can watch happen in the interactive tool.
Where a result is contested — and the superficial-alignment claim in Module 2 genuinely is — this course says so rather than picking the tidier side.
Stage 1 established that pretraining optimises one thing: the average negative log probability of the next token. Take that literally and base-model behaviour stops being strange.
Ask a faithful next-token predictor "What is the capital of France?" and a reasonable continuation is "What is the capital of Germany?" — because across a very large corpus, that string appears most often inside lists of quiz questions. The model is not confused, not broken and not failing. It is doing exactly what it was optimised to do, correctly.
Post-training is everything that converts that object into something that answers. It is conventionally three moves, and being precise about what each buys is most of the value of this stage.
| Stage | Teaches | Data | Cannot do |
|---|---|---|---|
| Instruction tuning (SFT) | Format and task shape: when you see a question, produce an answer. | Thousands to low hundreds of thousands of demonstrations. | Add knowledge. If it was not in pretraining, SFT teaches confident statement, not knowing. |
| Preference optimisation | Ranking among acceptable answers: which of two good replies is better. | Pairwise comparisons, human or model-generated. | Create a capability the base model lacks. It reweights what already exists. |
| RL with verifiable reward | Reaching a checkable right answer, and the search behaviour that finds it. | Problems with automatic verifiers — maths, code, tests, constraint satisfaction. | Work where correctness cannot be checked by a program. |
The InstructGPT paper (Ouyang et al., 2022) reports that in human evaluations on their prompt distribution, outputs from the 1.3B-parameter InstructGPT model were preferred to outputs from the 175B GPT-3 — a model with 100× more parameters.
Read that carefully, because it is easy to over-read. It does not say the small model knows more. It says that on this prompt distribution, judged by these labellers, a well post-trained small model produces more preferred outputs than a much larger base model. Knowledge did not move; usability did. That gap — between what a model contains and what you can get out of it — is the entire territory of this stage.
SFT is the same next-token cross-entropy loss on a different corpus, plus one thing that is easy to get wrong.
The corpus is demonstrations: a prompt and a good response, formatted with whatever chat template the model will use at inference. The change is loss masking — you compute the loss only over the response tokens, not over the prompt tokens.
If you do not mask, you are also training the model to generate prompts — to predict what a user would say. That is not a crash and not obviously visible in the loss curve. It is a diffuse, quiet degradation: capacity spent modelling a distribution you will never sample from, and a model slightly more inclined to continue conversations on the user's behalf. It is one of the most common bugs in hand-rolled SFT implementations precisely because nothing complains.
SFT is dramatically cheaper than people expect. A useful instruction-following model can be produced from thousands of examples, not millions, and past a certain point additional data of the same shape adds very little.
What matters more than count is coverage of the task distribution you will actually see. Ten thousand examples that all look like the same three tasks will produce a model that is excellent at those three and no better than the base model elsewhere. This is the practical form of the "quality over quantity" advice, and it is more actionable than the slogan: diversity of task shape is what you are buying, not prose polish.
LIMA (Zhou et al., 2023) trained on 1,000 carefully curated examples and reported strong instruction-following, motivating the superficial alignment hypothesis: a model's knowledge and capabilities are learned almost entirely in pretraining, and alignment mainly teaches it which sub-distribution of formats to use when talking to a user.
Evidence for. Token-distribution analyses — notably Lin et al. (2023) — find that a base model and its aligned version produce nearly identical distributions at most token positions, with the divergence concentrated on stylistic and discourse tokens: hedges, politeness markers, safety disclaimers, formatting. That is a real, specific, measured result and it points the same way.
Where it is over-read. "Alignment is superficial" gets used to mean post-training barely matters, which is not what the evidence supports. The token-distribution result is measured on particular models and prompt sets; it does not extend cleanly to safety behaviour under adversarial pressure, to multi-turn agentic use, or — most importantly — to the reasoning models of Module 7, where RL demonstrably produces behaviour that was not present in the base model's output distribution in any meaningful sense.
The defensible position: for instruction-following style, alignment does look largely superficial. For capabilities elicited by large-scale RL, it does not. Anyone stating the strong version flatly in 2026 is behind the evidence.
It goes like this. "Let's fine-tune the model on our documentation so it knows our business." It is proposed constantly, it is funded regularly, and it produces the same outcome every time: a model that has learned the register of your documents and now hallucinates in your house style, with more confidence than before, because it has learned that text which sounds like your documents is what it should produce.
You do not need an experiment for this. You have the loss function from Stage 1.
Fine-tuning minimises next-token cross-entropy on your corpus. That objective rewards producing text with the statistics of your documents. It contains no term for factual accuracy, no term for retrieval, and no mechanism by which a specific fact seen a handful of times becomes reliably recoverable from a specific query phrasing months later.
What gradient descent actually does with a rare fact seen three times in a 50,000-example fine-tuning set is nudge a very large number of parameters by a very small amount in a direction that makes those particular token sequences slightly more likely in that particular context. What it does much more strongly — because the signal is far denser — is learn your formatting, your vocabulary, your sentence rhythm and your typical topics. Style is dense in the data; individual facts are sparse. Gradient descent goes where the signal is.
The answer is not "never fine-tune." It is that fine-tuning is a behaviour instrument, and it is very good at it:
Every item on that list is behaviour. None of them is knowledge.
Two places, and only two. Pretraining — expensive, and almost certainly not yours to do. Or context at inference time — retrieval, tool calls, long context, a database query. That is what retrieval-augmented generation is for, and it is why RAG did not lose to long context: putting the fact in the context window is the mechanism, and how it gets there is an engineering choice.
When someone proposes fine-tuning to fix a factual problem, the useful move is not to say no. It is to ask a question that makes the mechanism visible:
"What do you expect the gradient to be teaching? If the answer is a format or a tone, this is the right tool. If the answer is a fact, the loss function has no term for that — and the failure mode is that we get a model that states our facts confidently whether or not it has them."
That reframes it from a disagreement about tooling into a question about the objective, which is both more persuasive and more likely to be right. Occasionally the answer is a format problem wearing a knowledge costume, and then you should fine-tune.
Step 1 is a deliberate design choice worth pausing on. Humans are bad at absolute ratings and good at comparisons. Ask ten people to score a response out of ten and you get ten different calibrations, drifting within each person over a session. Ask which of two is better and you get a far more consistent signal. Every method in this course inherits that decision, including DPO.
Maximise reward, but stay close to where you started. The second term is not regularisation in the usual sense. It is a leash, and what it is restraining is specific.
The reward model is a neural network trained on a finite sample of human comparisons. It is therefore an approximation of human preference, accurate in the region where it saw data and arbitrary outside it.
The policy is an optimiser, and it is very good. Given free rein it will find the inputs on which the reward model is most wrong — regions of output space no human ever rated, where the reward model confidently returns a high score for text a person would find worthless. This is reward hacking, and the outputs are often not subtle: degenerate repetition, weird formatting tics, or fluent nonsense with the surface features the reward model learned to like.
The KL penalty prevents this by making the policy pay for straying from the reference distribution, which is roughly the region where the reward model has evidence. So β is not a tuning knob for output quality — it is a trust parameter for your reward model. Lower it and you are asserting your reward model is accurate further out than it was.
The important part: this is structural, not a bug. Any learned proxy optimised hard enough will be exploited — it is Goodhart's law with a gradient. Which means the ceiling on classic RLHF is set by reward-model accuracy, and no amount of engineering removes it. Module 7 is about the field's answer, which was to stop learning the reward.
Three models resident at once — policy, reward model, frozen reference. An online loop that must sample from the policy at every step, so generation is inside the training loop. And enough hyperparameter sensitivity that reproducing somebody else's published result is a project rather than an afternoon.
It works, and it produced the first genuinely useful assistants. It is also exactly the kind of apparatus that invites someone to ask whether all of it is necessary. Somebody did.
Here is the summary worth being able to give verbatim, before the derivation makes it earned:
For the KL-constrained reward objective that RLHF optimises, the optimal policy has a closed form — it is the reference policy reweighted by the exponentiated reward. You can therefore invert that relation to express the reward as a function of the policy, substitute it into the preference likelihood, and optimise the policy directly on preference pairs with no reward model and no sampling loop.
The RLHF objective from Module 4 is not an arbitrary loss — it is a KL-regularised expectation, and that family has a closed-form maximiser. For every prompt x:
In words: the optimal policy is the reference policy, reweighted by the exponentiated reward, then renormalised. High-reward responses get their probability multiplied up; low-reward ones down; the reference distribution supplies the shape. Z(x) is the normalising constant that makes it sum to one, and it involves a sum over every possible response — so it is completely intractable to compute. Remember that; it is the crux.
Take logs and rearrange for r. Nothing subtle here, just algebra:
Read this as a statement about what a reward function is. Any reward function corresponds to some optimal policy, and any policy implicitly defines a reward — the log-ratio of its probability to the reference model's, scaled by β. The reward and the policy are two descriptions of the same object.
The intractable β log Z(x) is still sitting there, which is why this step alone is not enough.
The Bradley-Terry model says the probability a human prefers yw to yl is a sigmoid of the difference of their rewards:
This is the step. Substitute the expression from step 2 for both rewards. The term β log Z(x) depends only on x, and both responses share the same x — so it appears identically in both and cancels in the subtraction. The intractable quantity vanishes not because it was approximated or bounded, but because we only ever needed a difference.
If someone asks you where DPO's reward model went, this is the line to point at.
You now have the probability of a preference expressed purely in terms of the policy you are training and a frozen reference. Fit the policy by maximum likelihood over your preference dataset:
That is a plain binary classification loss over pairs. Two forward passes through the policy and two through the frozen reference per example. No reward model, no rollouts, no third trainable network, no RL loop. You can implement it in an afternoon and it will work.
The quantity in bold is worth naming. β log(πθ/πref) is the implicit reward — DPO's reward function, expressed through the policy's own weights. The reward model was not removed from the problem. It was reparameterised into the policy, which is a different and stronger claim than "eliminated."
This is why the clearing-test question "where is the reward model in your DPO implementation?" is a trick. The wrong answer is "there isn't one." The right answer is that it is the log-ratio between your policy and the reference, and that the trick was noticing this quantity is exactly what Bradley-Terry needs.
It also has a practical payoff: you can compute that log-ratio at any time to see how your policy scores its own outputs, which gives you a reward signal for monitoring without ever having trained one.
The paper is Rafailov, Sharma, Mitchell, Ermon, Manning and Finn (2023), and its abstract's central claim is that DPO matches or improves on PPO-based RLHF for sentiment control, summarisation and single-turn dialogue while being substantially simpler to implement and train. Note the scope of that comparison — it is the next module's subject.
PPO samples from the current policy, scores those samples, and updates. DPO optimises against a fixed dataset of pairs generated by something else — usually an earlier model or a different one entirely.
That matters because as training proceeds, your policy drifts away from the distribution that produced the data, and you are increasingly optimising preferences over responses your model would never actually generate. You are answering the question "which of these two things somebody else said is better" when the useful question is "of the things I am about to say, which is better."
This is the main reason on-policy methods retain an advantage on the hardest problems, and it is why the strong practical recipes iterate: generate with the current policy, get preferences on those samples, train, repeat. Iterative DPO recovers much of the gap at the cost of the simplicity that was the whole point.
Step 3 of the derivation assumed a specific model of human preference: that there exists a scalar reward per response, and that preference probability is a sigmoid of the difference. That is a real assumption, and it fails in ordinary ways.
Preferences that are intransitive (A over B, B over C, C over A — common with multiple annotators or multiple criteria) have no consistent scalar reward. Preferences that are near-deterministic — where one response is simply always better — push the sigmoid towards saturation, and since the only way to increase a saturating sigmoid is to increase the margin without bound, the optimiser will happily drive the policy arbitrarily far from the reference. IPO (Azar et al., 2023) was designed for exactly this failure.
This is the one that surprises people, and it is worth understanding mechanically because it changes how you build preference data.
DPO's loss increases the margin between chosen and rejected. Nothing in it requires the chosen response's absolute probability to rise. In practice both frequently fall, with the rejected falling faster — the margin improves exactly as the loss demands, while the model becomes less likely to produce either.
Razin et al. (2024) named this likelihood displacement and showed it can be catastrophic rather than cosmetic: probability mass shifts not into a neutral background but into responses of opposite meaning. Their example is the one to remember — training a model to prefer "No" over "Never" can sharply increase the probability of "Yes."
The mechanism follows from the gradient. The DPO update moves parameters along
∇logπ(yw) − ∇logπ(yl) — the
difference of two gradients. When chosen and rejected are semantically similar, those
gradients are nearly parallel, so their difference is small and points somewhere unrelated to either.
Take a step along it and the probability mass you removed from the rejected response does not
reliably land on the chosen one. It lands wherever that difference direction happens to point.
The practical consequence is concrete and actionable: preference pairs whose two responses are very similar are actively dangerous, not merely low-signal. If your rejected responses are minimally edited versions of your chosen ones — which is exactly what you get from a "generate then lightly corrupt" data pipeline — you are constructing the failure case. The interactive tool in Module 9 lets you watch this happen and find the boundary.
| Method | Fixes | Cost |
|---|---|---|
| IPO (Azar et al., 2023) | Saturation under near-deterministic preferences, by replacing the sigmoid objective with a bounded one. | Another hyperparameter; less widely used in practice. |
| KTO (Ethayarajh et al., 2024) | The requirement for pairs. Learns from unpaired good/bad labels, which is what production feedback actually looks like. | Discards the ranking information a pair carries. |
| ORPO (Hong et al., 2024) | The reference model, using an odds-ratio penalty folded into the SFT loss. Halves memory and merges two stages into one. | Loses the KL leash that the reference model provided. |
| SimPO (Meng et al., 2024) | Length bias and the reference model, using length-normalised average log-probability as the implicit reward. | Also gives up the reference anchor; more sensitive to its margin term. |
| Iterative / online DPO | The off-policy problem, by regenerating preferences from the current policy each round. | Reintroduces a generation loop — the thing DPO removed. |
Human annotators prefer longer responses, at a rate that is substantially independent of whether the extra length adds anything. That bias is in your preference data, so it is in your implicit reward, so it is in your policy. Every preference-optimised model drifts verbose, and the drift is not a bug in the algorithm — it is a faithful reproduction of the signal you gave it.
Two practical consequences. Always report response length alongside your win rate, because a win rate that moved together with a length increase has told you nothing you can act on. And when comparing methods, check whether the comparison controlled for length; a great many published improvements are partly length effects, which is why SimPO's length normalisation exists.
A neural network predicts what humans prefer. It approximates taste, and like any approximation it can be gamed: the policy finds where the reward model is wrong and exploits it.
This puts a hard ceiling on how far you can optimise, and Module 4 showed the ceiling is structural rather than fixable.
A program checks the answer. Did the arithmetic come out right? Do the tests pass? Does the proof check?
There is nothing to hack, because the reward is not a model — it is a fact. You can optimise hard and long without the signal degrading.
The term reinforcement learning with verifiable rewards was introduced in Tülu 3 (Lambert et al., Allen Institute for AI, 2024), which replaced the reward model in the standard RLHF objective with a verification function; the underlying idea also appears in DeepSeekMath's work from the same period. The change is small to describe and it removed the binding constraint on the entire approach.
Once the reward is a verifier rather than a critic, you can afford enormous numbers of rollouts, and the model can discover strategies that raise the probability of passing the check. What emerged — checking work, backtracking, trying a second approach, allocating more tokens to harder problems — was not in anyone's demonstration set.
DeepSeek-R1 (2025) is the cleanest public account. R1-Zero was trained with large-scale RL directly on the base model, with no SFT stage at all, and developed strong reasoning behaviour including what the authors describe as an "aha moment" — the model spontaneously learning to pause and re-evaluate its own approach.
R1-Zero also came out with poor readability and language mixing — the authors note the base model was trained on multiple languages, especially English and Chinese, and the RL-trained model would switch between them mid-reasoning. Optimising purely for a verifier optimises purely for the verifier: nothing in "is the final answer correct" rewards the trace being legible to a human.
That is why the shipped R1 is not R1-Zero. It adds a cold-start SFT stage on readable, well-formatted reasoning data before RL, then further rounds of rejection sampling and RL. Verified reward gave the capability; supervised data made it usable. Anyone telling you post-training is now just RL has skipped the stage that made the product shippable, and it is the same lesson as Module 6's length bias: you get precisely what you measured.
PPO needs a learned value network to estimate a baseline — the advantage is "how much better was this than expected," and something has to supply the expectation. That is a second network the size of the policy, trained alongside it.
GRPO (from DeepSeekMath, Shao et al., 2024) removes it with a substitution that suits the setting exactly. Sample a group of completions for the same prompt. Score them all with the verifier. Use the group's mean score as the baseline. Advantage becomes "how much better than my siblings was this attempt."
The reason this works here and not everywhere: you are already sampling many completions per prompt, because that is how RL with a verifier operates. The baseline is therefore free — it is a statistic over samples you had to generate anyway. One fewer network to hold, train and debug, and a variance-reduction story that is easy to reason about.
A capability is amenable to RL exactly to the degree that you can write a checker for it.
That is a sharper predictor of what models will get good at next than any parameter count. Domains with cheap, reliable verifiers — competition mathematics, code with tests, formal proofs, constraint satisfaction, anything with a compiler or an interpreter — race ahead. Domains where quality is a matter of judgement — taste, strategy, interpersonal nuance, most writing — move at the pace of preference data, which is slow and expensive.
Two cautions before you use this as a roadmap. Verifiers can be gamed too when they are weaker than they look: reward-hack a unit-test suite and you get code that passes tests and does not work, which is reward hacking with extra steps. And "we'll just write a verifier" is easy to say about domains where a good one is a research project in itself.
Question one: is this a knowledge problem? If the model does not know something, stop. Module 3 applies and nothing in this stage will help. Go to retrieval, tools or context. Answering this question honestly saves more money than every other decision here combined.
Question two: can you write a program that checks the answer? If yes, you are in the RLVR regime and you have the strongest option available. If no, you are in the preference regime and your ceiling is the quality of your preference data.
Those two questions eliminate most of the decision space. What remains:
| Symptom | Reach for | Roughly | Watch for |
|---|---|---|---|
| Ignores instructions; wrong output shape; won't produce valid JSON | SFT | 1K–50K demonstrations | Loss masking. Task-shape diversity, not example count. |
| Answers are acceptable but not good; tone or judgement is off | DPO or a successor | 5K–100K pairs | Length drift. Near-duplicate pairs causing displacement. |
| You have thumbs-up/down from production, not pairs | KTO | Whatever you have | Selection bias — who bothers to click. |
| Fails a task where correctness is checkable | RLVR with GRPO | 1K–100K problems with verifiers | Verifier gaming. Readability of the trace. |
| Memory-constrained, want SFT and preference in one pass | ORPO | Pairs, plus SFT data | No reference model, so no KL leash. |
| Prompt is long, expensive and repeated every call | SFT (distillation) | 10K+ input/output pairs from the long-prompt system | Nothing — this is often the highest-ROI fine-tune available and it is rarely the one proposed. |
Preference optimisation reweights among responses the model already produces. If the model does not produce good responses at all yet, there is nothing to reweight, and DPO on a base model gives you a base model with a slightly different distribution over documents.
SFT first, always, then preference optimisation. R1's own pipeline is the argument: even with RL strong enough to produce novel reasoning, they still put a supervised cold-start stage in front of it, because the RL had nothing to say about being readable.
The one legitimate exception is R1-Zero-style pure RL as a research result — demonstrating that RL alone can elicit reasoning. It is not the recipe for a product, and its authors did not ship it as one.
Describe a real problem. Commit to the method you think it needs before the tool says. The most useful outcome is the one where the tool tells you post-training cannot help, because that is the verdict that saves the quarter.
Module 6 claimed that DPO can reduce the probability of the response you preferred, and that similarity between chosen and rejected is what drives it. This is that claim, running.
A deliberately minimal model: three possible responses —
chosen, rejected, and one other response the model could give.
DPO steps along ∇logπ(chosen) − ∇logπ(rejected); the
sliders set how those gradients relate. Presets first, then move things.
It is a toy. Three outcomes, not a vocabulary; one scalar similarity, not a real gradient geometry; a fixed step size instead of an optimiser. It cannot predict what your run will do.
What it is faithful to is the mechanism. The update direction really is
∇logπ(yw) − ∇logπ(yl); the log-probability
of the chosen response really does change by its own logit's movement minus the probability-weighted
average movement of every outcome; and the condition under which the chosen response's probability
falls really does come out as a competition between (1−s) — how much useful signal
survives the subtraction — and pother · α — how much mass
the third response absorbs. Displacement in real models is this, in many more dimensions.
The prediction the toy makes that is worth taking seriously: as similarity approaches 1, the
useful signal vanishes faster than the leakage does. That is why near-duplicate preference pairs are
a data-construction hazard rather than merely a weak signal, and it is checkable in your own runs by
logging logπ(yw) rather than only the margin.
Take a small base model and make it measurably better, twice, by two different mechanisms. First write your own SFT loop — not TRL's — and produce an instruction-following version. Then write your own DPO loss from the paper and improve it further on a preference set you constructed, with a held-out split you never trained on.
Report a number with an honest error bar and the mean response length alongside it. Then answer these unaided:
You have cleared Stage 3 when you have written the DPO loss by hand, watched it move a real model, and can defend the derivation at a whiteboard.
Not the test: running a TRL example script. TRL is excellent and you should use it in production. The point of writing the loss once is that when a run misbehaves you are debugging code you understand rather than a library you trust — and Module 6 is a catalogue of the ways these runs misbehave while the loss curve looks fine.