AI modelsDive 02 of 06
From base model to assistant: post-training
Follow one coding request through chat templates, supervised fine-tuning, preference learning and reinforcement learning, down to the log-probabilities and memory that make it work.
- Levels
- 9
- Reading
- 40 min
- Depth
- Idea to Machine
- Published
One prompt, three possible answers
“Write average(xs). Reject an empty list.”
Ticks mark the reference: 70%, 28%, 2%. Blue bars show the optimized policy.
Illustrative finite policy, computed as π*(y) ∝ πref(y) exp(r(y)/β). These are invented answers and rewards.
Knowing the pattern is not following the request
You ask a language model: Write a Python function average(xs). Reject an empty list.
It has seen Python, division, exceptions and thousands of explanations of averages. Yet knowing those patterns does not determine what it should write next. It could answer you, repeat the question, imitate a forum discussion or invent the next exercise in a textbook. Each is a plausible continuation of some document.
A A pretrained language model before assistant-specific adaptation. It models continuations of its training distribution. learns to continue text. It can answer questions, especially with suitable prompting, but its objective does not appoint it the assistant in your conversation. Post-training changes that distribution: follow the request, use the requested format, admit uncertainty where appropriate, stop after the answer. The deployed model still predicts tokens. Its weights now make a different set of continuations likely.
Here is the difference we want. Both sides are invented examples, not outputs measured from a named model. The right side follows the empty-list requirement as well as the easy part, computing a mean.
Possible base continuation
Exercise 4. Write a Python function average(xs). Reject an empty list. Exercise 5. Modify your function to accept weights.
Desired assistant answer
def average(xs):
if not xs:
raise ValueError("empty list")
return sum(xs) / len(xs)Three different sources of supervision
You can teach that behavior by writing the answer yourself. This is supervised fine-tuning, or SFT: the training example contains the desired assistant response. You can also show two candidate answers and choose the better one. Preference learning turns those comparisons into a signal that is cheaper to provide than a perfect rewrite. Finally, you can run a checker. A unit test supplies feedback without asking a person whether the code looks convincing.
These signals solve different problems. A demonstration specifies what to imitate. A preference distinguishes two attempts. A verifier checks a defined property of an attempt. An assistant might need all three. A test can establish that an empty list raises an exception; it cannot by itself judge whether an explanation is respectful, whether a follow-up question is useful, or whether the user wanted a different API.
The influential InstructGPT paper, Ouyang et al. (2022), describes a sequence of demonstrations, comparison-based reward learning and reinforcement learning. It builds on Christiano et al. (2017), who trained reward predictors from human comparisons of behavior. Those are published recipes, not a specification of every assistant sold today. Later work changes the objectives, feedback sources and order of stages.
Post-training can use a small fraction of pretraining compute while strongly changing visible behavior. There is no universal fraction: repeated rollouts, long answers and multiple training rounds can be expensive. The point is where the starting capability comes from. You begin with weights that already represent language and code, then spend a more targeted budget making those capabilities useful under an interaction contract.
A running example and a notation
Keep the average(xs) request in mind. A correct, concise answer guards the empty list. A plausible but incomplete answer omits the guard. An exploitable answer adds polished praise around broken code. Our toy reward will sometimes like the third answer best. That deliberate mistake makes it possible to see the difference between optimizing a score and improving an assistant.
We will write the prompt as x, the response as y, and the policy as πθ(y | x), the probability assigned by the trainable language model. πref is a frozen reference policy, usually a checkpoint chosen before the preference or RL stage. A reward r(x, y) is a scalar score. The same notation will carry through PPO, DPO and GRPO.
The machinery of loss, backpropagation, Adam and distributed training already exists. So does the token-by-token generation loop. Post-training joins those two systems around a more selective definition of success. The first engineering question is deceptively small: which bytes belong to the assistant's answer?
Post-training changes which continuations the model prefers. The supervision can come from an example, a comparison or a checker; each defines success differently.
A conversation becomes a training example
Your application sends messages with roles. The transformer receives one sequence of token IDs. A A serialization rule that turns role-labelled messages into the token sequence expected by a model. bridges the two: write a conversation-start marker, a role header, message content and an end-of-turn marker for each turn. At inference time, leave the final assistant turn open so generation continues inside it.
The exact serialization is part of a checkpoint's interface. Here we use an explicitly generic template: [begin], [system], [user], [assistant] and [end]. These illustrative names stand for boundary tokens in our diagram; actual checkpoints define their own spellings. An actual tokenizer may encode its special markers as individual reserved IDs, while ordinary message content splits into subword tokens.
For training, append the desired answer and its end marker. For generation, append only the assistant header after the user's message. That distinction is essential. If your prompt ends with a completed assistant turn, the next likely text may be another user turn. If you duplicate a conversation-start marker or use a different header convention, you have changed the input distribution before any sampling setting comes into play.
The context mask and the loss mask answer different questions
The model learns the correct guard by reading the user's requirement as context and predicting the demonstrated answer. A loss mask selects assistant targets while retaining system and user tokens as context. In the visualizer, blue underlines mark positions whose token prediction contributes to the objective. Toggle the mask to see what changes.
- Target: loss weight 1
- Context only: loss weight 0
Generic template with word-sized display pieces, not a vendor tokenizer. The Python body belongs inside def average(xs).
The causal attention mask determines which earlier positions a token can read. The padding mask identifies batch filler. The assistant loss mask determines which target predictions you score. They are three separate objects. With user loss weight zero, attention still reads the user's tokens. Gradient from an assistant prediction flows through computations that processed the prompt.
For a multi-turn example, you may supervise every assistant turn or only the last one. That is a dataset choice. For tool use, you also need a decision about tool-call syntax and tool-result roles. The same principle holds: serialize exactly what the deployed model will see and choose targets according to what you want it to generate. Boundary tokens are behavior too, especially the decision to end a response.
Templates also make stopping explicit. Include the assistant's end marker among the supervised targets so the model can learn where its turn finishes. At deployment, the generation engine recognizes the appropriate marker and returns control to the application. A model that writes a useful answer and then fabricates a user reply has learned some content but failed the interaction contract. A stopping rule can cut that reply off, while correctly templated demonstrations teach the model to end its own turn.
What a good SFT record contains
Our record needs more than a natural-language question and some Python. It needs a clear contract: an empty input raises ValueError, normal numeric inputs return their arithmetic mean, and the answer should be a function. A demonstration that silently returns zero on empty input teaches the wrong contract even if it is beautifully formatted. Quality begins before the optimizer touches the batch.
Deduplicate related examples and split by problem family before evaluation. A renamed copy of average(xs) in the test set tells you little about generalization. Include awkward cases and varied formulations, so the assistant learns a requirement rather than a memorized phrase. Decide how to handle contradictory instructions, underspecified requests and tasks it cannot complete. Those examples help define the assistant's behavior at the edges of the dataset.
Long conversations make this harder. Truncation can remove the user's original requirement while leaving the answer that depends on it. Packing can place two unrelated conversations side by side. Keep the turn boundaries, supervision spans and attention boundaries coherent; otherwise a valid-looking token tensor can represent an impossible interaction. We will return to the actual shifted tensors at the bottom of the dive.
Version the template with the dataset and the checkpoint. If a later tokenizer revision changes how a header is encoded, the same visible conversation can become a different training stream. Save token counts and assistant target counts during preprocessing, then inspect representative decoded records. A suspiciously small assistant target count often reveals truncation or boundary mistakes before an expensive training run. Compare training and serving serialization on the same conversation as a simple integration check.
Once the template and mask are settled, SFT is ordinary language-model training on selected positions. Politeness and the empty-list guard both become token predictions whose probability the loss pushes upward.
Teach by example, update fewer weights
SFT uses the same cross-entropy objective as pretraining. Let s be the entire serialized stream and mt its binary target mask. The normalized loss for a batch is:
LSFT(θ) = − Σt mt log πθ(st | s<t) / Σt mt
The sum runs over shifted target positions across the batch. Training uses the actual preceding tokens from the demonstration, a practice called teacher forcing. At each selected position, it asks how much probability the model assigned to the next demonstrated token. The demonstration supplies the answer directly; executing candidate functions becomes useful when we introduce verifiable rewards.
That explains both SFT's strength and its limit. A small collection of careful answers can make a useful interaction pattern easy to follow. But the loss treats a demonstrated variable name, a stylistic phrase and a crucial empty-list guard as next-token targets. It has no built-in distinction between harmless alternatives and semantic mistakes. Two equally correct implementations can have different token losses, while plausible code with one fatal error can look locally easy to predict.
Token-normalized loss also gives long responses more total weight than short ones. You can instead normalize within each example and average examples, but that changes the objective. Neither choice excuses leaving it implicit. A run that suddenly favors verbosity may reflect the data mixture or normalization rule before it reflects anything about a preference algorithm.
LoRA changes what can move
Full fine-tuning updates the pretrained matrices. LoRA, Hu et al. (2021), freezes a matrix W and learns an additive low-rank update. For a linear layer with input width din and output width dout, use A with shape r × din and B with shape dout × r:
W′ = W + (α/r) BA
W: dout × din · A: r × din · B: dout × r
Trainable parameters: r(din + dout), versus din·dout
Rank r limits the rank of this update, not the rank of the whole pretrained matrix. The scale α/r controls the update's magnitude under the original LoRA convention. A common initialization makes one factor zero and the other random, so the initial effective layer equals W while gradients can begin moving the factors. Both factors zero would stall learning at initialization.
For a 4096 × 4096 projection at rank 16, the adapter has 131,072 parameters against 16,777,216 in W: 128 times fewer trainable numbers for that layer. A model-wide count depends on exactly which projections you target. Adapting Q and V is a small budget; adapting all attention and MLP projections is a larger one. There is no target list that is best for every model and task.
- Trainable parameters
- Adapter training state
Square attention projections; three gated MLP matrices. Adapter state: 16 bytes/parameter, excluding base weights and activations.
The calculator assumes square Q, K, V and O projections and a gated MLP with three matrices. Grouped-query attention uses narrower K and V matrices, so use the actual checkpoint shapes for a production count. Frozen base weights still occupy memory, and backward propagation still traverses the network to reach the adapters. LoRA saves trainable state while retaining the forward and backward work through the transformer.
At inference, you can merge BA into a compatible full-precision W, avoiding a separate adapter path. Keeping adapters separate makes switching tasks convenient but adds work and deployment choices. Merging into a quantized base can involve dequantization and requantization, with rounding effects that deserve evaluation.
QLoRA saves the frozen base as well
QLoRA, Dettmers et al. (2023), stores the frozen pretrained weights in a 4-bit representation and backpropagates into LoRA adapters. Computation dequantizes weight blocks as needed. The paper combines NF4, double quantization and paged optimizers. Four bits describes the base representation, not the precision of every multiplication, activation, gradient or optimizer state.
The distinction matters when estimating memory. You save base storage and adapter optimizer state, but activations still grow with batch size and context. The quantization dive explains NF4 and QLoRA's representation. Here the useful separation is between an objective, such as SFT or DPO, and a parameterization, such as full weights or LoRA. You can combine them. The objective determines which feedback the adapters learn from.
SFT selects the target tokens. LoRA selects the trainable degrees of freedom. QLoRA also compresses the frozen base; these choices operate on different parts of training.
Learn what a better answer means
SFT gives the model one answer to imitate. Now sample two answers to our request. One computes a mean but fails on empty input; the other includes the guard. A reviewer can choose the second without writing a new function. Store the prompt x, the preferred response yw and the rejected response yl. Repeat this over many prompts and candidate pairs.
A reward model converts those comparisons into a scalar rφ(x, y). For a language model backbone, a scalar head can read a designated final response representation. The model sees the prompt as well as the response, because the same code can be correct for one contract and wrong for another. The score is learned; there is no intrinsic unit called “one helpfulness.”
The Bradley–Terry paired-comparison model (1952) assigns odds using exponentiated scores. In the reward parameterization used here, with σ(z) = 1/(1 + exp(−z)):
P(yw ≻ yl | x) = σ(rφ(x, yw) − rφ(x, yl))
LRM(φ) = − E(x,yw,yl)∼D log σ(rφ(x,yw) − rφ(x,yl))
If the preferred answer's score is much lower, the loss is large. If its score is much higher, the pair supplies little remaining gradient. Adding the same prompt-dependent constant to both scores changes nothing: comparisons identify score differences. A reward value of seven is not a calibrated probability of correctness, and it cannot be compared across independently trained reward models without further assumptions.
What the reviewer accidentally teaches
A reviewer may favor longer answers because they usually contain more useful details. The reward model can learn length as a shortcut. That is reasonable on its training comparisons and dangerous when an optimizer begins searching for the highest score. Length is now a feature the policy can increase without adding a guard or repairing any code.
The trainer below has only two features: helpfulness h and normalized length ℓ. Its score is wh·h + wℓ·ℓ. The table shows the same four candidate pairs under both labeling policies. Balanced labels prefer the more helpful candidate. Length-biased labels prefer the longer candidate, flipping the choices on pairs 2 and 4 while every feature stays fixed. Train with each policy to isolate what the labels teach.
| Pair | Candidate A (h, ℓ) | Candidate B (h, ℓ) | Preferred |
|---|
- Helpfulness weight wh
- 0
- Length weight wℓ
- 0
- Pair loss
- 0.693
Linear reward r = wh·h + wℓ·ℓ; batch gradient descent, learning rate 0.15, L2 coefficient 0.02. Changing preference labels resets training; candidate features stay fixed.
The L2 term keeps this small separable dataset's weights from growing without bound. The toy exposes named features that a neural reward model would learn implicitly. In a real pipeline, examine length-controlled comparisons, held-out preference accuracy and examples optimized specifically to fool the scorer. A randomly sampled validation set may never contain the strange answers a policy learns to produce.
This is the practical shape of Goodhart's problem: once a proxy becomes the optimization target, the optimizer can push it into regions where its relationship with what you wanted breaks down. An answer full of agreeable phrases may score well while its function divides by zero. The model need not intend deception for the optimization to discover an exploitable correlation.
Human and AI feedback
Humans do not need to supply every comparison. Constitutional AI, Bai et al. (2022), describes a supervised phase using AI critiques and revisions guided by principles, followed by reinforcement learning from AI-generated preference feedback. This is one published RLAIF recipe: reinforcement learning from AI feedback.
The source of a label and the algorithm that consumes it are separate choices. AI preferences can train a reward model; preference pairs can also train DPO. A constitution supplies criteria, while a judge applies them imperfectly. Judge errors, shared blind spots and style preferences still enter the resulting dataset. Audit what the labels reward, whether they come from people or an AI judge.
For our function, a reviewer rubric might explicitly prioritize the guard, numerical correctness and an answer that fits the request. Those criteria make disagreements more diagnosable. They also expose when a scalar is collapsing genuinely different goals: completeness, brevity and readability can conflict. Treat the reward as a training instrument: measure its behavior against the task requirements.
Optimize a reward without losing the assistant
With a trained reward model, you can generate answers, score them and update the policy to make successful answers more likely. That is reinforcement learning from human feedback when the reward traces back to human preferences. The score is not differentiable through the sampled text. Policy gradients instead use the log-probability of the actions that produced that text.
Optimizing only the reward invites the flattering, broken function from our hero. Add a cost for leaving a frozen reference distribution. For complete responses, the KL-regularized objective is:
J(θ) = Ex∼D [Ey∼πθ(·|x) rφ(x,y) − β DKL(πθ(·|x) ∥ πref(·|x))]
DKL(πθ ∥ πref) = Ey∼πθ log[πθ(y|x)/πref(y|x)]
β is nonnegative and measured relative to the reward scale. With positive β, probability assigned to unusual answers has a cost. The reference is commonly an SFT checkpoint that already behaves usefully. It supplies a statistical anchor, not a proof of safety or correctness. If a bad answer is common in the reference, anchoring can preserve it too.
Four model roles and two different comparisons
A conventional PPO implementation has a trainable policy, frozen reference, frozen reward model and trainable value model. The value model estimates expected future reward from a prefix. Implementations can share backbones or use different sizes; four roles need not mean four identical independent allocations. The old policy πold is the policy that generated the current rollout. Saving its action logprobs often avoids keeping another full old-policy model resident.
Keep πold and πref distinct. PPO's importance ratio compares the new policy to the recent rollout policy. The KL term compares it to the longer-lived reference. One limits how aggressively you reuse a batch; the other expresses what objective you want to optimize.
Prompt x → sampled answer y
One score for the answer
Token logprobs for the same answer
Token rewards → residuals δt → advantages Ât
Clipped policy update + value regression
- Snapshot the policy as πold and sample answers. Save the sampled token logprobs; they are the denominator in PPO's ratio.
- Evaluate the frozen reward and reference models. Give each response token a KL-shaped reward; add the learned score at the last response token.
- Use the value predictions and temporal-difference residuals to estimate which actions beat their expected returns.
- Reuse this rollout for a few minibatch passes. The ratio compares the updated policy with πold, while the KL anchor still compares with πref.
- Fit values to return targets, synchronize the updated policy with the rollout engine, and collect fresh answers. The reference stays fixed.
Turn a response score into token-level credit
At response position t, let ht be the prompt plus the generated prefix and at the chosen token. A common reward shaping rule gives a token reward −β log[πold(at | ht)/πref(at | ht)] and adds rφ(x,y) at the final response token. In expectation over rollouts from πold, the sum of these log ratios estimates the sequence KL for that policy. Individual sampled terms can be negative.
The value prediction Vψ(ht) estimates the remaining shaped return. Generalized advantage estimation, Schulman et al. (2015), builds an advantage from temporal-difference residuals. For a T-token response indexed from t = 0, use token reward ut, discount γ and a terminal value of zero:
δt = ut + γ Vψ(ht+1) − Vψ(ht)
Ât = Σl=0T−t−1 (γλ)l δt+l
The finite sum stops at the response boundary. γ is often one for episodic language tasks; λ trades variance against dependence on imperfect value predictions. A positive advantage means the sampled token led to a better return than predicted at that prefix. The estimator propagates terminal rewards backward to assign token-level credit.
Why clip the probability ratio?
PPO, Schulman et al. (2017), reuses rollout data for several optimization steps. The action ratio ρt corrects for changes from the behavior policy. Its clipped surrogate, which we maximize, is:
ρt(θ) = πθ(at|ht) / πold(at|ht)
Lclip(θ) = Et min[ρt Ât, clip(ρt, 1−ε, 1+ε) Ât]
For Ât = 2 and ε = 0.2, raising the ratio from 1 to 1.2 raises the term from 2 to 2.4; raising it to 1.5 brings no further benefit for that sampled term. For a negative advantage, the other side clips. Taking the minimum matters: changes that move the policy in the wrong direction still receive a penalty. Clipping is not a hard constraint on the whole policy's KL.
The lab uses an exact expectation over three candidate responses, with four clipped-surrogate gradient steps per update. A separate full-distribution KL term makes the anchor easy to inspect. Set β to zero and run several batches. The high-reward broken answer takes over, while measured quality falls. Reset and repeat at β = 3 to see how the same reward flaw is restrained.
- Expected flawed reward
- Expected true quality
- KL to reference
- Flawed reward (0–3)
- True quality (0–1)
Three-action bandit; exact expected clipped surrogate, ε = 0.2, four gradient steps per rollout, step size 0.25. KL uses the full categorical distribution.
Notice that β must be interpreted together with the reward's units. Multiplying every reward by ten while keeping β fixed makes the anchor relatively weaker. Reward centering, scaling and advantage normalization therefore deserve explicit documentation. When comparing runs, preserve those conventions or report the change. A slider value copied from another experiment is not a portable measure of how much the policy may drift.
A production token policy has many more actions, learned value estimates and noisy minibatches. It also needs value regression and careful normalization. The toy removes those complications while keeping a genuine clipped objective. Its lesson survives the simplification: improving an imperfect reward more efficiently can make the actual result worse. Stronger optimization and a better objective are different achievements.
Put the preference loss inside the policy
PPO trains a reward model and then searches for a policy that maximizes it under a KL anchor. Direct preference optimization, Rafailov et al. (2023), rewrites that problem so the preference likelihood can update the policy directly. The algebra connects the reward objective to a likelihood over paired preferences.
First solve the regularized policy problem
Fix a prompt x and reward r(x,y). Assume β > 0, reference support wherever the policy may put mass, and a finite normalizer. Define Z(x) by summing the tilted reference probabilities. Then the regularized objective can be written as a constant minus a KL:
Z(x) = Σy πref(y|x) exp(r(x,y)/β)
π*(y|x) = πref(y|x) exp(r(x,y)/β) / Z(x)
Ey∼π r(x,y) − β DKL(π ∥ πref)
= β log Z(x) − β DKL(π ∥ π*)
The last KL is minimized at zero, so π* is the optimal unrestricted policy for this fixed reward. Our hero computes precisely this solution over three responses. Real language models do not enumerate every possible response or evaluate Z. The closed form is useful because we can invert it:
r(x,y) = β log[π*(y|x)/πref(y|x)] + β log Z(x)
Now take a Bradley–Terry comparison at the same prompt. The prompt-only β log Z(x) appears in both rewards and cancels. Replace π* with the trainable πθ and fit the observed comparison labels. Writing the policy's implicit reward as r̂θ(x,y) gives:
r̂θ(x,y) = β log[πθ(y|x)/πref(y|x)]
LDPO(θ) = − E(x,yw,yl)∼D log σ(r̂θ(x,yw) − r̂θ(x,yl))
This implicit reward chooses one representative from the rewards that differ by a prompt-dependent constant. Its difference is all the preference likelihood needs. You evaluate policy logprobs on both recorded responses, compare them with frozen reference logprobs and backpropagate the classification loss. There is no separately trained reward or value model and no generation inside the basic offline update.
What β changes
Let Δθ = log πθ(yw|x) − log πθ(yl|x), and define Δref the same way. The per-pair loss is softplus(−z), where z = β(Δθ − Δref). Its derivative with respect to Δθ is −βσ(−z). The explorer shows both the loss and the positive magnitude of that derivative.
- z = β(Δθ − Δref)
- DPO loss
- Gradient weight βσ(−z)
Relative margin Δθ − Δref: −16 to +16 · loss: 0 to 33
Margins are log π(winner) − log π(loser), using summed response logprobs. The gradient with respect to Δθ is negative.
In the fixed-reward optimum, increasing β keeps π* closer to the reference. In DPO's fitted classifier, β also scales the logit and its gradients. At equal margins, a larger β increases the gradient magnitude. For a fixed desired classifier logit, a larger β needs a smaller relative policy margin. Thus β changes both the desired policy shift and the gradient scale.
β = 0 is a degenerate DPO limit: the classifier logit is always zero and the gradient vanishes. With small positive β, the resulting policy still depends on data coverage, learning rate, optimization steps and the reward scale implicit in the labels.
Offline data and online exploration
Basic DPO uses a fixed comparison dataset. It teaches the policy about recorded winners and losers. It may generalize to new responses, but it does not collect evidence about its own newly preferred outputs during that update. An online method refreshes candidates or feedback as the policy changes. PPO does this through rollouts; an iterative preference recipe can generate fresh pairs and use DPO again.
Why train on the loser too?
If you discard the rejected answer and perform SFT on the winner, the model learns to reproduce that winner's tokens. DPO instead asks whether the winner gained enough probability relative to the loser and relative to the reference. The rejected response tells the optimizer what distinction the comparison was about. For our request, a loser that differs only by the missing guard supplies a sharper contrast than an unrelated bad paragraph.
A DPO update can improve the margin by reducing the loser's probability more than the winner's, even when both absolute probabilities fall. Language-model normalization couples every response through shared parameters. Track chosen and rejected logprobs as well as the margin, and evaluate generated answers rather than interpreting a decreasing preference loss as a complete account of behavior.
Sequence sums preserve the probability-ratio derivation. Dividing each response's logprob by its length creates a different score, so the original derivation no longer applies unchanged. Length differences can matter in both the labels and the objective; changing the arithmetic is a modeling decision to evaluate, not a neutral correction.
The derivation provides a reward-to-policy correspondence under its assumptions. A finite neural model still has to learn from limited preference data, including any bias in its labels. If long, broken functions win the comparisons, direct optimization can learn the same preference without ever allocating a standalone reward model.
Our empty-list example makes the practical choice concrete. If you have a reliable offline set that distinguishes correct guards from missing ones, DPO is a simple training path. If the policy keeps inventing new ways to satisfy a superficial judge, collecting fresh feedback may be more useful than taking more epochs over the old pairs. Match the source of evidence to the behavior you need to correct.
| Method | Evidence per update | Additional learned component |
|---|---|---|
| SFT | Recorded target answer | None |
| PPO + reward model | Fresh scored rollouts | Reward and value models |
| Offline DPO | Recorded preference pairs | None; reference logprobs required |
| GRPO + verifier | Fresh groups of checked answers | None; reference policy required |
DPO substitutes an implicit reward into the Bradley–Terry likelihood. The partition function cancels within a prompt, leaving a direct loss on policy-versus-reference logprob margins.
When the answer can be checked
Our request is unusually convenient: important parts of success can be executed. A unit test can check the returned mean and the empty-input exception. That gives a reward grounded in a checker rather than a learned prediction of what a reviewer would prefer. Reinforcement learning with verifiable rewards uses this kind of signal to train on tasks such as code, mathematics and formal proofs.
A verifier is only as complete as the property it checks. Testing average([2, 4]) == 3 rewards a function that always returns three. Add negative inputs, singleton lists and the required exception. Even then, passing these tests is evidence about these cases, not a proof for every input. Keep evaluation tests separate from the training verifier, and execute generated programs in an isolated environment with time and resource limits.
assert average([2, 4]) == 3
assert average([-3, 3]) == 0
assert average([7]) == 7
try:
average([])
except ValueError:
pass
else:
raise AssertionError("empty input must raise ValueError")A group can supply the baseline
DeepSeekMath, Shao et al. (2024), introduced group relative policy optimization, GRPO. Sample G responses for the same prompt from the old policy, score each response and compare its reward with the group's rewards. For outcome supervision, a sequence's normalized advantage can be assigned to each of its response tokens:
ȓ = (1/G) Σi=1G ri
s = √[(1/G) Σi=1G (ri − ȓ)2]
Âi,t = (ri − ȓ) / s
This page uses the population standard deviation and defines the advantage as zero when s = 0. Implementations may use an epsilon or a different standard-deviation convention. A binary group [0, 0, 1, 1] has mean 0.5, standard deviation 0.5 and advantages [−1, −1, 1, 1]. A perfect group [1, 1, 1, 1] supplies no within-group contrast, just like a group in which every answer fails.
Population standard deviation within this group. Equal rewards give zero advantages by convention; finite inputs are clamped to 0–10.
DeepSeekMath subtracts β·KL directly from the maximized surrogate, outside the rewards used to compute group advantages. Its sampled penalty is k3, introduced below. For response i with Ti tokens, define the token ratio and clipped term as:
ρi,t(θ) = πθ(ai,t | hi,t) / πold(ai,t | hi,t)
Ci,t(θ) = min[ρi,t Âi,t, clip(ρi,t, 1−ε, 1+ε) Âi,t]
JGRPO(θ) = Ex∼D, y1,…,yG∼πold [(1/G) Σi=1G (1/Ti) Σt=0Ti−1 (Ci,t(θ) − β k3,i,t(θ))]
Here ai,t is token t in response i, and hi,t is its prompt and generated prefix. Each k3,i,t compares the policy with the reference at that prefix. The KL term acts on the objective separately from the raw rewards ri that determine Âi,t.
GRPO removes the learned value network by estimating a baseline from the sampled group. It still needs a policy, rollouts, old action logprobs and usually a frozen reference. With a learned scorer it also needs that scorer.
Group normalization changes the weighting of prompts. A small amount of reward variation can be amplified to unit scale; no variation gives no direction at all. The proportion of mixed-success groups matters. Prompts that are always solved or always failed may be poor sources of this particular signal even if they matter greatly to users.
Reasoning traces and compute at test time
The January 2025 DeepSeek-R1 report describes R1-Zero, trained with RL directly from a base model, and R1, which adds cold-start data and further stages. In the R1-Zero experiment, the authors report growing response lengths and behaviors such as checking and reconsidering solutions. They also report readability problems and language mixing. These are observations about that published experiment.
A plausible mechanism is straightforward: if additional generated work helps an answer pass the checker, outcome rewards can favor policies that perform that work. Check longer traces against actual outcomes and measure their cost as well as their success rate. Establishing whether a visible explanation faithfully reflects internal computation requires separate evidence.
Test-time compute is the inference budget you spend after training. You might generate one longer attempt, sample several independent attempts, or use a verifier to select among them. Those choices spend tokens differently. Under an idealized independent-success model with per-attempt success probability p, the chance that at least one of N attempts succeeds is 1 − (1 − p)N. You still need a way to identify that success; majority voting and a reliable verifier solve different selection problems.
For the average function, testing four candidates and returning the first that passes is a concrete use of extra inference. For an open-ended explanation without a decisive checker, choosing among four polished answers is less settled. The inference dive covers generation and the KV cache; here the key consequence is that training can change how much useful work a policy performs within the generation budget.
A trainer attached to an inference fleet
Online RL attaches a trainer to a generation system. For our request, the rollout engine first generates candidate functions. A scorer or verifier evaluates them, and the trainer consumes tokens, old logprobs, masks and rewards. After updates, the rollout engine needs the new weights. That loop mixes workloads that prefer different hardware schedules.
Generation spends time in sequential decoding and holds KV caches for unfinished responses. Training evaluates known token sequences in parallel, retains activations for backward and updates optimizer state. A configuration that runs the forward-and-backward pass efficiently may leave too little memory for a productive rollout batch. Conversely, a large rollout pool can fill the queue faster than the trainer consumes it.
Responses · old logprobs
Tokens · masks · rewards · version k
Reference logprobs · advantages · updates
Weight synchronization: trainer → generation engine
Count roles before counting GPUs
A useful first estimate assumes separate, equal-size models. PPO holds two trainable models, policy and value, plus two frozen models, reference and reward. Offline DPO holds a trainable policy and a frozen reference; it can sometimes precompute the reference logprobs and release that reference during training. GRPO with a programmatic verifier holds a trainable policy and frozen reference. A learned GRPO scorer adds another model.
A trainable parameter in the calculator uses 16 bytes: two for a BF16 weight, two for its BF16 gradient, four for an FP32 master weight and eight for Adam's two FP32 moments. A frozen BF16 parameter uses two bytes. That is a stated storage convention; mixed-precision and optimizer implementations differ. For equal-size P-parameter models, this gives PPO 36P bytes, DPO 18P and verifier-based GRPO 18P.
- Resident weight copies
- Weights
- Gradients + optimizer
- Total state
Equal-size models, BF16 weights/gradients, FP32 master weights and Adam moments; aggregate GiB before sharding. Add activations, KV caches, rollout engines and workspaces separately.
At seven billion parameters, these assumptions give about 234.7 GiB for PPO and 117.3 GiB for DPO or verifier-based GRPO before activations and rollout caches. Those totals cover the system; per-GPU peaks also depend on placement and sharding. Sharding, offloading, shared backbones, smaller scorers and adapter training change placement and storage. An independently resident generation replica adds its own weights.
Old rollout logprobs are much smaller than a policy copy: one scalar per sampled token. Likewise, reference logprobs for a fixed DPO dataset can be cached rather than recomputed every epoch. But a changing reference, different tokenization or altered truncation invalidates the corresponding cache. Store the provenance that makes the numbers meaningful.
Where throughput goes
Long answers occupy decode slots for longer, consume more KV memory and create longer training examples. Grouped rollouts multiply that cost. Variable completion lengths make a synchronous batch wait for its slowest members unless scheduling is careful. A code verifier can become the bottleneck when tests are slow, even if model inference is fast.
Weight synchronization also has a price. If the trainer and rollout engine are separate, transferring large weight tensors consumes bandwidth and creates a version boundary. Asynchronous collection can overlap work, but increases policy lag. The saved πold logprobs identify the behavior distribution. Large policy lag can still make these samples poor training data. Track policy versions and choose an explicit freshness policy.
Useful measurements include generated tokens per second, scored responses per second, training tokens per second, the distribution of answer lengths, verifier latency, synchronization time and task success per total compute budget. A single utilization percentage hides the shape of the loop. Improve the stage that limits useful completed updates, while keeping the feedback and sampling distribution coherent.
The details of optimizer sharding, memory and parallelism belong to the training stack. Post-training adds a scheduling problem: keep inference, feedback and learning supplied with compatible batches. Even a mathematically correct objective can be trained badly if the reward arrives for the wrong completion or the logprobs belong to the wrong policy version.
PPO, DPO and GRPO differ in model roles and data flow as well as loss functions. Count training state, rollout state and synchronization separately.
The tensors underneath the objectives
All the objectives above eventually read the same primitive: the log-probability of the token that actually occurred. A forward pass returns logits with shape [batch, length, vocabulary]. At each prediction position, normalize across the vocabulary, gather the target token's entry and mask away unwanted targets.
Use log-space arithmetic. For logits zj, compute log πj = zj − logsumexp(z). A stable implementation finds a = maxj zj, then evaluates logsumexp(z) = a + log Σj exp(zj − a). Exponents are now at most one. Logits [1000, 1000] give logprobs [−log 2, −log 2], without first overflowing exp(1000).
Gathering selects one vocabulary column per position using the observed target ID, avoiding a full one-hot tensor. Summing those gathered logprobs over the assistant response gives log πθ(y | x), including its chosen ending convention. DPO uses the sums for both responses; PPO needs the individual action logprobs for its token ratios.
import torch
import torch.nn.functional as F
# ids, attention_mask, assistant_mask: [batch, length]
# logits: [batch, length, vocab], before the causal shift
logits = model(ids, attention_mask=attention_mask).logits
logp = F.log_softmax(logits[:, :-1, :].float(), dim=-1)
targets = ids[:, 1:]
chosen = logp.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
mask = assistant_mask[:, 1:].bool() & attention_mask[:, 1:].bool()
sequence_logp = chosen.masked_fill(~mask, 0).sum(-1)
sft_loss = -chosen.masked_select(mask).mean()
# Require at least one assistant target before taking the mean.The one-position shift is the contract
The logits at stream position j predict the token at j + 1. Therefore the selected logits slice is [:, :-1, :], and the targets and target masks are [:, 1:]. The assistant header can have loss weight zero while its final position still predicts the first assistant body token, whose target mask is one. Masking the predictor positions instead would drop or misassign boundary targets.
| Predictor token | Target token | Target mask | Contribution |
|---|---|---|---|
| last user token | [assistant] | 0 | Context |
| [assistant] | def | 1 | First answer target |
| def | average | 1 | Answer target |
| last answer token | [end] | 1 | Learn to stop |
| [end] | [pad] | 0 | Ignore |
Padding fills physical batch storage around the actual conversation. It must be absent from attention keys where appropriate, sequence logprob sums, token reward reductions and advantage statistics. Packed examples additionally need the intended attention isolation and position handling. Reject or deliberately skip examples with no supervised targets before taking a mean; an empty reduction can quietly become a non-finite loss.
The code uses FP32 for log-softmax arithmetic and keeps logits in log space. For summation it replaces masked entries with zero, rather than relying on multiplication that could encounter zero times negative infinity. Real training implementations may fuse the normalization and gather to avoid materializing a large FP32 [batch, length, vocabulary] tensor. The mathematical selection is the same.
Three estimates of the same KL
Let p be the current policy and q the reference at one fixed prefix. Draw an action a from p, and define ℓ = log p(a) − log q(a), u = q(a)/p(a) = exp(−ℓ). Schulman's note on approximating KL divergence compares estimators conventionally called k1, k2 and k3. With this direction and sampling distribution:
k1 = ℓ
k2 = ½ℓ2
k3 = u − 1 − log u = exp(−ℓ) − 1 + ℓ
The mean of k1 is DKL(p ∥ q), though an individual k1 can be negative. k2 is nonnegative and locally approximates KL near p = q; it is generally biased. k3 adds the control variate u − 1, whose expectation under p is zero when the distributions have common support. The inequality u − 1 ≥ log u makes k3 nonnegative on each sample, while preserving the correct expectation.
- Exact DKL(p ∥ q)
- Sample mean k1
- Sample mean k2
- Sample mean k3
A two-outcome distribution with common support. k1 and k3 are unbiased under policy sampling; k2 is a local approximation.
Common support matters: if q assigns mass where p is exactly zero, the simple control-variate expectation changes. The lab keeps both outcomes strictly positive. It uses expm1(−ℓ) + ℓ for k3 to improve accuracy near zero. Very large ratios can still overflow, so production code must monitor finite values rather than assuming a nonnegative formula is automatically numerically safe.
Sampling direction matters just as much. Tokens generated by πold follow that behavior distribution, even after πθ changes. If you replace logprobs in an estimator without correcting the sampling distribution, its expectation can change. An unbiased gradient calculation must also account for how the sampling distribution depends on the parameters. Autodifferentiating a scalar KL estimate through fixed samples alone can give a biased gradient.
At a fixed prefix, exact vocabulary KL sums over every possible next token. A sampled estimate reads only the generated action's logprob under each model, saving storage and work. Across a whole response, summing conditional log ratios along sampled prefixes recovers sequence KL in expectation under the generating policy. That is why the high-level regularizer ends up as per-token arithmetic.
Our average function has now crossed every layer: a request became a template, a mask selected its answer, an objective compared examples or returns, and a trainer reduced gathered logprobs. None of those layers knows that an empty-list guard is important by itself. The demonstrations, comparison labels and verifier make it important, and the tensor conventions determine whether that signal reaches the intended predictions.
The behavior you teach depends on the feedback you choose. The behavior you actually train depends on a shifted target, a mask and a correctly oriented log-probability ratio.
End of Post-training · 9 levels, Idea to Machine
Where to next
Next in AI models · 03 of 06
Mixture of experts, from router to all-to-all
Follow a token into a sparse expert layer: router scores, balance and capacity, expert parallelism, token permutation, and the memory traffic behind conditional computation.