AI modelsDive 01 of 06

How a neural network learns, from loss to all-reduce

Turn prediction errors into gradients, build a tiny autograd engine, and train a real network in your browser. Then follow one weight update through optimizer memory, GPU collectives, pipeline schedules and the compute budget of a language model.

Levels
9
Reading
45 min
Depth
Idea to Silicon
Published

A prediction error, drawn as a landscape

Vertical: rug’s logitHorizontal: mat’s logit

Teach a model to choose “mat”.

Start with the wrong prediction. Follow 40 computed gradient steps as two learned logits change.

Update
0
Target probability
1.0%
Loss + penalty
5.047

Toy model: logits [mat, rug, moon] = [x, y, 0]. Objective: −ln p(mat) + 0.04(x² + y²); SGD rate 0.35. Contours join equal losses.

Level 01/09Idea

Give the model something to get wrong

You give a language model “the cat sat on the” and want it to put a high probability on “mat”. A freshly initialized model has no reason to do that. Its matrices contain small random numbers, chosen to keep signals in a useful numerical range. The architecture supplies a way to compute predictions. Training supplies the values that make those predictions useful.

The inference dive follows a token through a model whose weights already exist. Here we follow the other direction: a known next token becomes an error signal, that signal becomes a change to every relevant weight, and a cluster repeats the operation over a large corpus. The forward computation is familiar. What makes training different is the machinery wrapped around it.

Write the model as f(x; θ). The input x is the context. The parameters θ are all the learned numbers: matrix entries, embedding vectors, normalization scales and any biases. During a forward pass, the input changes while the parameters stay fixed. Between passes, the optimizer changes the parameters. That separation is the central rhythm of training.

Turn a prediction into a scalar

Our miniature vocabulary has three possible next tokens: mat, rug and moon. Suppose the model predicts probabilities [0.5, 0.3, 0.2] and the observed next token is mat. The For a one-hot target, the negative natural logarithm of the probability assigned to the observed class. is −ln(0.5) ≈ 0.6931 nats. If it assigns mat only 0.1, the loss is about 2.3026. Giving the observed token a lower probability costs more.

A second training example might have rug as its target with probability 0.25. Its loss is −ln(0.25) ≈ 1.3863. The mean of the two losses is about 1.0397 nats. Equivalently, their joint likelihood is 0.5 × 0.25 = 0.125, and the mean loss is −ln(0.125) / 2. Taking logarithms turns a product over many observations into a sum that is easier to compute and differentiate.

The model emits unrestricted scores called logits, not probabilities. Softmax converts each logit into pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ). Subtracting the largest logit before exponentiating gives the same result and avoids unnecessarily large exponentials. Libraries usually compute cross-entropy directly from logits using a stable log-sum-exp calculation.

Context: “the cat sat on the” · target: “mat”

mat
—
rug
—
moon
—
Loss (nats)
1.099
∂L / ∂z_mat
−0.667

Three-token vocabulary; the other two logits stay at zero. Probabilities use stable softmax.

Raise the target’s logit and watch its probability rise and its loss fall. The displayed gradient is computed from this exact three-token distribution.

For softmax cross-entropy, the derivative with respect to logit zᵢ is pᵢ − yᵢ, where the one-hot target y is 1 for mat and 0 elsewhere. At equal logits, every probability is one third. The target gradient is therefore −2/3; the other two are +1/3. Subtract the gradient and you raise the target score while lowering the others. We have a direction to move before we have learned anything about the internals of the network.

Training uses the observed prefix at every position, a setup often called teacher forcing. In “the cat sat on the mat”, the position after cat predicts sat using the observed tokens the cat. It does not have to sample a correct sat before the next position can train. With a causal transformer, many positions can be evaluated together in one forward pass. Inference generates an unknown continuation sequentially; training already has that continuation available as targets. The mask preserves the prediction problem while letting the hardware process a whole sequence at once.

A real next-token objective repeats this at every unmasked position in every sequence. The input token at one position is paired with the token immediately after it as a target. A causal attention mask prevents a position from reading its own answer. Padding and any excluded positions must be left out of both the loss sum and its normalization.

The target is an observed continuation, not a certificate that every other continuation is wrong. Many contexts admit several reasonable answers. Across examples, minimizing expected cross-entropy pushes the model toward the data’s conditional token distribution. It also rewards reproducing errors, biases and accidental patterns in that data. Choosing the dataset and choosing the objective are part of choosing what the system learns.

Learning begins with a measurable error. Cross-entropy makes a missed prediction into a scalar whose derivatives tell the model how to change.

Check yourselfThe target has probability 0.25. What is its next-token cross-entropy?

Level 02/09Interface

The loop you actually write

You now need to repeat that calculation and change the weights. The basic interface fits in a short PyTorch loop: forward, loss, backward, optimizer step, clear gradients. Below is a complete CPU example. Its tiny model predicts each next token from the immediately preceding token, so it learns a transition rule rather than attention or long-range context.

The synthetic stream cycles through integers 0 to 7. Each input is a 16-token window and each target is shifted by one position. This makes the code runnable without a dataset download. The validation portion uses the same simple periodic rule, so a low validation loss here establishes that the rule was learned; it says little about generalization to natural language.

A complete next-token training loop · PyTorch
import math
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(7)
# A toy token stream: predict the next integer modulo 8.
tokens = torch.arange(8193) % 8
windows = tokens.unfold(0, 17, 16)
train = TensorDataset(windows[:400, :-1], windows[:400, 1:])
valid = TensorDataset(windows[400:, :-1], windows[400:, 1:])
loader = DataLoader(train, batch_size=16, shuffle=True)
model = nn.Sequential(nn.Embedding(8, 32), nn.Linear(32, 8))
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3)
steps = 10 * len(loader)
warmup = 20
step = 0

for epoch in range(10):
    model.train()
    for x, target in loader:
        if step < warmup:
            factor = (step + 1) / warmup
        else:
            progress = (step - warmup) / max(1, steps - warmup - 1)
            factor = 0.1 + 0.9 * (1 + math.cos(math.pi * progress)) / 2
        for group in optimizer.param_groups:
            group["lr"] = 3e-3 * factor

        logits = model(x)                    # [batch, time, vocabulary]
        loss = nn.functional.cross_entropy(
            logits.flatten(0, 1), target.flatten()
        )
        loss.backward()
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)
        step += 1

    model.eval()
    with torch.no_grad():
        x, target = valid.tensors
        validation_loss = nn.functional.cross_entropy(
            model(x).flatten(0, 1), target.flatten()
        )
    print(epoch, float(loss.detach()), float(validation_loss))
Forward: logits
Loss: one scalar
Backward: gradients
Step: new weights
Zero: clear gradients
  1. Read a batch and run the model. Autograd saves the intermediate values its backward rules need.
  2. Compare logits with the target token IDs. Average the loss over the batch and token positions.
  3. Backpropagate from the scalar loss. Gradients accumulate in each parameter’s grad field.
  4. The optimizer reads the gradients, updates its state and changes the parameters.
  5. Clear the gradients so the next batch starts fresh. Intentional accumulation clears only after several microbatches.
1 / 5
The five operations in one optimizer update. Intermediate values belong to the forward/backward pair; optimizer state survives across updates.

Batches, steps and epochs

A batch is a group of examples used to estimate one gradient. Averaging 16 sequences of length 16 means the loss here averages 256 token predictions. An optimizer step changes the parameters once. An epoch is a pass over the training dataset. Those units answer different questions: how noisy is this gradient, how many changes have been made, and how often has each training example been revisited?

The example has 400 training windows and 25 batches per epoch. Ten epochs produce 250 optimizer steps. On a streaming corpus, an epoch may be an inconvenient or ambiguous unit, so training schedules often count processed tokens or optimizer steps instead. Always state which count controls the learning rate.

PyTorch adds into a parameter’s .grad field. It does not overwrite that field on every backward pass. The loop clears gradients after each update; clearing before the next forward is also common. With gradient accumulation, you deliberately run several forward/backward pairs before a step. Divide each equal-sized microbatch’s mean loss by the number of microbatches so the accumulated gradient remains a mean. Unequal token counts require weighting by the number of valid targets.

At cluster scale, keep the batch units explicit. If each of 8 data-parallel ranks processes 2 sequences per microbatch and accumulates 4 microbatches, one optimizer update covers 64 sequences. At 2,048 valid tokens each, that is 131,072 target tokens. Increasing accumulation changes how many tokens contribute to an update without increasing the activations stored for a single microbatch. It also changes the number of updates made over a fixed token budget. A schedule measured in updates must account for that change.

Keep one set of answers out of training

The training loss tells you how well the model fits examples it is allowed to learn from. Validation loss estimates performance on held-out examples. For text, split by document or another meaningful source boundary before making overlapping windows. Otherwise, near-duplicates can cross the split and make memorization look like generalization. A final test set should stay out of repeated design decisions.

model.eval() changes layers such as dropout to their evaluation behavior. torch.no_grad() prevents recording the backward graph. They solve different problems, which is why validation uses both. If training loss keeps falling while validation loss rises, the run is overfitting. More steps can make that gap worse. Better data, regularization, an earlier stopping point or a smaller model may help; inspect the split before blaming the optimizer.

Change the step size over the run

A learning-rate schedule makes early and late updates different. Warmup increases the rate over the first few updates. It gives the run a gentler start while optimizer statistics and activations settle. After warmup, cosine decay reduces the rate smoothly toward a chosen floor. The code uses a floor of ten percent of the peak, rather than quietly training the last steps at zero.

Start · warmupTraining progressEnd · floor
A representative warmup plus cosine schedule: rise to the peak over the first tenth of the run, then decay to 10% of that peak.

Warmup is a technique, not a guarantee of stability. A bad peak rate can still break the run after warmup ends. The next level opens the optimizer to see why the size and direction of an update matter so much.

Level 03/09Runtime

Choose the next step

Imagine freezing the batch and varying the weights. Each choice produces a loss, so the loss is a surface over parameter space. A two-weight model gives you a surface you can draw. A billion-weight model gives you the same mathematical object in a space you cannot picture. The gradient gₜ = ∇θ L(θₜ₋₁) contains one partial derivative per parameter, evaluated at the current weights.

The derivative is local. For a small displacement δ, the first-order approximation is L(θ + δ) ≈ L(θ) + gᵀδ. Choosing δ = −ηg makes the linear term −η‖g‖², which is negative for a nonzero gradient and positive rate η. A finite step can still increase the true loss because the approximation ignores curvature.

Three ways to use the gradient

Stochastic gradient descent uses θₜ = θₜ₋₁ − ηₜgₜ. “Stochastic” refers to estimating the objective’s gradient from a sampled batch. You can evaluate the full-dataset gradient, but doing so before every update is often more expensive than making many useful approximate steps.

Momentum keeps a velocity: uₜ = μuₜ₋₁ + gₜ, then θₜ = θₜ₋₁ − ηₜuₜ, starting at u₀ = 0. This is the unnormalized velocity convention used in our demos. Another convention multiplies the new gradient by 1 − μ; its rate needs a corresponding rescaling. Gradients that keep pointing the same way accumulate. Gradients that alternate across a narrow valley partly cancel.

  • SGD
  • Momentum (0.9)
  • Adam

f(x, y) = ½(x² + 20y²), start (−1.8, 1.3), 60 updates. Paths stop at the plot edge if they leave its bounds.

Compare actual update trajectories in a ravine. Increase the learning rate past 0.1 to make plain SGD unstable along the steep direction.

Adam keeps two exponentially weighted averages for each parameter: a first moment of the gradient and a second moment of its square. With zero initial state, its standard update is:

mₜ = β₁mₜ₋₁ + (1 − β₁)gₜ

vₜ = β₂vₜ₋₁ + (1 − β₂)gₜ²

m̂ₜ = mₜ / (1 − β₁ᵗ),   v̂ₜ = vₜ / (1 − β₂ᵗ)

θₜ = θₜ₋₁ − ηₜ m̂ₜ / (√v̂ₜ + ε)

Every operation is elementwise. The first moment supplies a smoothed direction; the second supplies a scale. Bias correction compensates for starting the averages at zero. With a first gradient of 1 and β₁ = 0.9, the raw first moment is only 0.1, but its corrected value is 1. The second moment is an average of squared gradients, not a centered variance. See Kingma and Ba’s Adam paper for the algorithm.

Step
0
Weight θ
1.00000
First moment m
0
Second moment v
0
Corrected m̂
0
Corrected v̂
0

Apply a gradient to see the bias-corrected update.

β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸. Plain Adam here; weight decay λ = 0.

Feed gradients to one parameter and inspect the actual Adam state. Try several positive gradients, then reverse the sign and watch the first moment take time to turn.

Where AdamW changes the equation

Adding an L2 penalty λ‖θ‖²/2 to the objective adds λθ to its gradient. In Adam, that term then enters the moment estimates and adaptive denominator. Loshchilov and Hutter’s AdamW decouples decay from that calculation. With moments computed from the data-loss gradient, the update used here is θₜ = (1 − ηₜλ)θₜ₋₁ − ηₜm̂ₜ/(√v̂ₜ + ε). The shrinkage acts directly on the previous weight.

Biases and normalization parameters are often placed in a group without decay; that is a model configuration choice. AdamW does not decide which tensors should be regularized. It gives you a rule whose decay strength is separate from the gradient’s coordinatewise rescaling.

Gradient-norm clipping is another control you may see between backward and the optimizer. Compute the global gradient norm and, if it exceeds a threshold c, multiply every gradient by c divided by that norm. This preserves direction while limiting the size of a gradient spike. It does not repair an incorrect loss normalization or make an excessive learning rate harmless. With distributed sharding, the norm must account for the complete intended gradient, rather than treating each shard’s local norm as the whole model’s norm.

The learning rate still matters enormously. An adaptive denominator does not make arbitrary step sizes safe. In the ravine above, the steep coordinate has curvature 20. SGD’s update there multiplies the coordinate by 1 − 20η, so convergence needs 0 < η < 0.1. The numerical boundary follows directly from the update, not from the appearance of the plot.

Level 04/09Runtime

Send responsibility backwards

The optimizer needs a gradient for every weight. Computing each one by perturbing that weight and rerunning the model would take billions of forward passes. Backpropagation gets all of them by reusing the structure of the computation. The key is that a network is built from operations whose local derivatives you already know.

Take one neuron: a = w × x, z = a + b, h = tanh(z). If a downstream loss sends back ∂L/∂h, the chain rule gives ∂L/∂z = (∂L/∂h)(1 − h²). Addition passes that incoming derivative to both inputs. Multiplication sends x × ∂L/∂a toward w and w × ∂L/∂a toward x. Each operation needs only its own inputs or output and the derivative arriving from later in the graph.

This is Differentiate a scalar result by traversing its computation graph in reverse, accumulating sensitivity at each input.. Backpropagation applies it to a neural network. Rumelhart, Hinton and Williams (1986) demonstrated how backward error propagation lets hidden units learn useful representations. The backward pass is a systematic chain-rule calculation; the optimizer is a separate algorithm that decides how to use the result.

Why derivatives add

Consider y = x × x + x. There are three routes from x to y: the left multiplication input, the right multiplication input and the final addition. At x = 3, the contributions are 3, 3 and 1, so the total derivative is 7. Assigning a gradient instead of adding it loses contributions. A shared weight in a transformer can influence the loss through many positions and paths; the same accumulation rule handles all of them.

Each row names its input nodes. Forward reveals one value; backward applies one local pullback. Finite differences use h = 10⁻⁵.

Walk a real scalar graph forward, then apply its local derivative rules in reverse. Switch to the shared-input expression to see why gradients must accumulate.

The graph lab uses a small TypeScript engine. Each Value stores its number, parents, gradient and a pullback: a closure that distributes its incoming gradient to its parents. A depth-first traversal visits each node once and records parents before children. Reversing that order ensures a node receives all downstream contributions before its pullback runs.

Here is the complete scalar engine from src/dives/training/math.ts, including tanh and the graph traversal, followed by a shared-input example.

The complete scalar autograd engine used by the graph lab
export class Value {
  data: number;
  grad = 0;
  label: string;
  parents: Value[];
  pullback: () => void = () => {};

  constructor(data: number, label = "", parents: Value[] = []) {
    this.data = data;
    this.label = label;
    this.parents = parents;
  }

  add(other: Value) {
    const out = new Value(this.data + other.data, "+", [this, other]);
    out.pullback = () => {
      this.grad += out.grad;
      other.grad += out.grad;
    };
    return out;
  }

  mul(other: Value) {
    const out = new Value(this.data * other.data, "×", [this, other]);
    out.pullback = () => {
      this.grad += other.data * out.grad;
      other.grad += this.data * out.grad;
    };
    return out;
  }

  tanh() {
    const out = new Value(Math.tanh(this.data), "tanh", [this]);
    out.pullback = () => {
      this.grad += (1 - out.data ** 2) * out.grad;
    };
    return out;
  }

  tape() {
    const order: Value[] = [];
    const seen = new Set<Value>();
    const visit = (node: Value) => {
      if (seen.has(node)) return;
      seen.add(node);
      node.parents.forEach(visit);
      order.push(node);
    };
    visit(this);
    return order;
  }

  backward() {
    const order = this.tape();
    order.forEach((node) => {
      node.grad = 0;
    });
    this.grad = 1;
    order.reverse().forEach((node) => node.pullback());
  }
}

const x = new Value(3, "x");
const y = x.mul(x).add(x);

y.backward();
console.log(y.data);  // 12
console.log(x.grad);  // 7

What the tape remembers

In eager PyTorch, operations on tensors requiring gradients construct a backward graph as the forward runs. Backward functions save tensors they need. A multiply may need its inputs; tanh can use its output. This recorded graph is often called a tape, though it is a dependency graph with branching and sharing rather than a simple list.

The scalar engine reads the saved parent values directly, so changing a value after forward and before backward would invalidate its calculation. Real autograd systems track in-place mutations and reject many unsafe changes. The familiar rule “do not update weights until backward finishes” preserves the relationship between the forward values and their derivatives.

Finite differences provide an independent check: ∂f/∂x ≈ [f(x + h) − f(x − h)] / (2h). The graph lab computes both answers. A very large h adds approximation error; a very small h subtracts almost equal floating-point values and amplifies rounding error. Smooth points and moderate h make useful test cases. A numerical check at one point cannot establish correctness for every operation and branch.

The chain rule also explains vanishing and exploding gradients. Along a long path, local derivatives multiply. Repeated factors smaller than one can make an early layer’s signal tiny; repeated large factors can amplify it. Our tanh example shows this directly when its output saturates. Residual connections add paths that carry a signal around a transformation, while initialization and normalization help keep activation and derivative scales useful. These architectural choices affect whether optimization has a usable signal before any optimizer hyperparameter can help.

For a scalar loss, reverse mode gets all parameter derivatives in one backward traversal. Dense linear layers typically do two matrix multiplies in backward for the one in forward. That makes backward roughly twice the forward arithmetic and the whole forward/backward pair roughly three times forward. It is an estimate for this workload, not an identity for every program. Backward also moves saved tensors and may recompute them.

Backpropagation reuses local derivatives and adds contributions where paths meet. One reverse traversal yields all the gradients of a scalar loss.

Check yourselfFor y = x × x + x at x = 3, what must backpropagation accumulate at x?

Level 05/09Runtime

Train a network here

You have the complete mechanism now. A forward pass produces a prediction; a loss compares it with an answer; backward computes gradients; an optimizer changes weights. The experiment below runs that mechanism on your CPU, in this tab. No predictions or gradient steps come from a server.

The dataset contains two interleaved spirals. Given a point’s x and y coordinates, predict which spiral it belongs to. A straight line cannot separate the two arms. The model needs a curved decision boundary, which makes it easy to see hidden layers learning something beyond a linear classifier.

The network has two tanh hidden layers followed by one output logit. A sigmoid turns that logit into the probability of class 1. Its binary cross-entropy is −y ln(p) − (1 − y) ln(1 − p). The implementation evaluates an equivalent stable expression directly from the logit. This is the two-class counterpart of the next-token objective: increase the probability of the observed answer.

Decision surface · x, y ∈ [−1, 1]

Cross-entropy · y = 0 to 1.5 nats

Steps 0 to 0

  • Train · solid
  • Validation · dashed

Class 0: amber. Class 1: teal. Background intensity shows confidence.

Updates
0
Train loss
—
Validation loss
—
Validation accuracy
—

Ready. Start training or train 50 batches manually.

2 → width → width → 1, tanh hidden units; binary cross-entropy, Float64 arithmetic. Fixed seed, 320 training points and 80 held-out points; batches sample with replacement.

Train a real two-hidden-layer MLP. The decision surface and losses update from its changing weights; outlined squares are held-out validation points.

Read the experiment

Start with Adam, width 16, batch 32 and a rate of 0.02. Train for several hundred updates. The initial surface is a poor guess; as the loss falls, the boundary bends around the spirals. A filled point is a training example. An outlined square belongs to validation and never participates in a gradient update. The loss plot uses a solid training line and a dashed validation line, so the distinction survives without color.

The model has 3w + (w + 1)w + (w + 1) = w² + 5w + 1 parameters for hidden width w, including biases. Width 16 gives 337 learned numbers. Each hidden unit computes a weighted sum and a tanh. The second hidden layer combines those responses, giving the output a vocabulary of curved features to use. There is no explicit spiral formula inside the model.

Reduce width to 4 and reset. Fewer parameters make fitting harder, though initialization and optimization still matter. Switch to SGD at the same numerical rate. It may move much more slowly: Adam’s normalized update has a different scale from SGD’s raw-gradient update. Comparing algorithms at one shared rate is an experiment, not a fair benchmark of their best settings.

Increase the batch from 8 to 128. Each update now averages more sampled points and generally estimates the data gradient with less sampling noise. But an update also performs more arithmetic. Compare runs at equal numbers of examples processed as well as equal optimizer steps. The demo samples with replacement, which keeps the batch rule simple; it does not implement shuffled epochs like the PyTorch example.

Useful ways to break it

Try a high learning rate and watch for oscillating or rising loss. On this small bounded-activation network, a bad rate can also push hidden units into tanh saturation. Tanh’s derivative is 1 − h²; when h is near ±1, very little signal flows backward through that unit. A network can stop improving without producing an infinity or a visible numerical explosion.

Watch the gap between training and validation instead of chasing the lowest training number. Our points share a simple generating distribution and the dataset is small, so this is a gentle generalization test. The validation curve can wiggle because updates improve some regions while hurting others. It is evaluated on all 80 held-out points each time the display records a measurement.

The trainer uses explicit dense-array backpropagation for speed. It applies the same chain rule as the scalar engine, but computes an entire layer’s derivatives together. The scalar graph is useful for inspecting responsibility; arrays avoid allocating a new object for every multiply in every batch. Large training frameworks take this further by expressing those layerwise derivatives as matrix kernels.

Continuous training runs in short requestAnimationFrame batches and pauses when the widget leaves the screen or the tab is hidden. With reduced motion enabled, use “Train 50 batches” for discrete updates. Changing width or optimizer resets the seeded model and its optimizer state; changing rate or batch size continues the current run. You can therefore test a new rate midway through learning without losing the weights.

The weights in a trained model are the accumulated result of many small, data-dependent updates. The browser experiment performs the same basic loop as a cluster.

Level 06/09System

Keep the update in memory

The browser model fits in a few kilobytes. Scale its parameter count to billions and a new question dominates: which values must remain available between operations? Training keeps much more than the weights needed for inference. Some state lives for the whole run; other state lives only until a backward operation consumes it.

The persistent-state bill

The classic mixed-precision Adam accounting in Rajbhandari et al.’s ZeRO paper is 16 bytes per parameter: 2 for a half-precision weight, 2 for its gradient, and 12 for optimizer state. Those 12 are a 4-byte FP32 master weight, a 4-byte first moment and a 4-byte second moment. In symbols, 2 + 2 + (4 + 4 + 4) = 16.

Seven billion parameters therefore need 112 GB, about 104.31 GiB, of persistent state when fully replicated. The half-precision weights alone need only 14 GB. This is why an inference-sized model can fail to fit for training before you have stored even one activation.

StateBytes / parameterWhy it survives
Low-precision weights2Inputs to the next forward pass
Low-precision gradients2Accumulate until the optimizer step
FP32 master weights4Retain small updates before recasting
FP32 first moment4Adam’s smoothed gradient history
FP32 second moment4Adam’s squared-gradient history

That 16-byte total names a specific layout. Some BF16 recipes keep FP32 gradients, some store FP32 parameters and cast selected operations, and some optimizers use different state formats. Pure FP32 Adam also totals 16 bytes per parameter: 4 for weights, 4 for gradients and 8 for moments, without a separate master copy. Same total, different contents. Inspect the implementation before treating a byte count as universal.

The forward values waiting for backward

Activations can be just as important. The backward rule for a matrix multiply needs forward operands, so a long network leaves a trail of saved tensors. Their size grows with microbatch size, sequence length, hidden width and depth. Materialized attention score matrices add a quadratic sequence-length term; memory-efficient attention kernels avoid retaining that full matrix.

Activation checkpointing stores selected boundaries and reruns the missing forward work during backward. Checkpointing each block can keep a block input instead of all its intermediate tensors. You trade more arithmetic for less live memory, and must reproduce random operations such as dropout consistently during recomputation. This is different from saving a durable training checkpoint to resume after a failure.

Persistent / rank
—
Saved activations
—
Total modeled
—

Activation sketch: L·B·S·H·a·8 bytes, or (L + 8)·B·S·H·a with block recomputation; a = 2 mixed, 4 FP32. Excludes attention matrices, live gathers, workspaces and allocator headroom. Width and layers are independent inputs.

Compare persistent state and saved activations as you change sharding, microbatch size and activation checkpointing.

Why fewer bits need more care

Mixed precision uses low-precision arithmetic where it is efficient and wider precision where range or accumulated error matters. Micikevicius et al. (ICLR 2018) describe FP16 training with FP32 master weights and loss scaling. A master copy preserves weight changes that would round away if each update were applied directly to a half-precision weight.

IEEE FP16 has 5 exponent bits and 10 explicit fraction bits. Its smallest positive normal is 2⁻¹⁴, its smallest subnormal is 2⁻²⁴, and its largest finite value is 65,504. BF16 has 8 exponent bits and 7 fraction bits: approximately FP32’s exponent range but coarser spacing near one. The quantization dive explains those layouts and the distinction between range and precision.

A gradient of 10⁻⁸ rounds to zero in FP16. Multiply the loss by 1,024 before backward and every gradient scales by the same factor. The scaled value can survive the low-precision calculation; divide the gradients by 1,024 in wider precision before applying the optimizer. Scaling after a gradient has already underflowed cannot recover it. BF16’s wider range usually removes the need for FP16-style loss scaling, though it still rounds values.

Original gradient
—
Direct FP16
—
Scaled FP16
—
Unscaled in FP32
—

IEEE binary16 round-to-nearest, ties-to-even, with subnormals. Hardware that flushes subnormals to zero loses more small values.

Quantize an actual small gradient to IEEE FP16, with and without scaling. Increase both controls enough and you can also produce overflow.
Check yourselfSeven billion parameters use ZeRO’s 16-byte mixed-precision Adam layout. How much persistent state is replicated on one GPU?

GPU memory still has ordinary allocation costs and fragmentation beneath these tensors. The memory dive follows that lower-level machinery. Here the next escape hatch is to distribute the persistent state instead of storing an identical full copy on every GPU.

Level 07/09System

Make many GPUs agree

Start with the simplest distributed arrangement: give each GPU an identical model and a different slice of the global batch. Each computes local gradients. Before the update, combine them into the gradient of the global batch. If every replica begins with identical weights and optimizer state and applies the same update, the replicas remain synchronized.

This is data parallelism. For equal-sized local batches whose losses are means, average the local gradients. With unequal numbers of valid tokens, weight by token counts instead. Averaging means with different denominators changes the objective. Data parallelism increases throughput, but ordinary replication does nothing to reduce the model-state memory required on one GPU.

A sum without a central collector

An all-reduce combines a buffer across ranks and leaves the result on every rank. A ring implementation splits each buffer into P chunks for P ranks. During reduce-scatter, each rank repeatedly sends a chunk to its neighbor and adds an arriving chunk. After P − 1 rounds, every chunk has been reduced and a different rank owns each complete sum. During all-gather, another P − 1 rounds distribute those complete chunks to everyone.

Transfer rounds
0 / 6
Bytes sent per rank
0 × message

Each message has N scalar chunks. GPU r starts with chunk c = r + 1 + c. Every rank sends to (r + 1) mod N simultaneously.

Follow real elementwise sums through reduce-scatter and all-gather. Each GPU sends one chunk per round; the final state contains the full sum on every GPU.

If the full gradient buffer has M bytes, each rank sends 2(P − 1)M/P bytes and receives the same amount. At P = 8 that is 1.75 times the full buffer sent per rank. For a 14 GB low-precision gradient buffer, the send volume is 24.5 GB. This is a per-rank transfer count; summing sends and receives would double it.

A simple time model is 2(P − 1)α + 2(P − 1)M/(P·B), with per-round startup α and effective link bandwidth B. Patarasuk and Yuan (2009) analyze bandwidth-optimal ring all-reduce. Rings are useful for large buffers; small buffers, network topology and in-network reduction can favor other algorithms. NVIDIA’s NCCL test documentation explains the corresponding all-reduce bandwidth factor.

Floating-point sums depend on order. Changing rank counts or collective algorithms can change the final low bits of a gradient. Synchronous distributed training aims to preserve the intended update mathematically, but does not automatically promise bitwise-identical runs across layouts.

Remove replicated state

ZeRO keeps the data-parallel computation while distributing state ownership. Stage 1 shards optimizer state; stage 2 also shards gradients; stage 3 also shards parameters. The DeepSpeed ZeRO guide describes those stages. With the mixed-precision layout above and P data-parallel ranks, the persistent bytes per parameter per rank follow directly:

LayoutWeightsGradientsOptimizerTotal
Replicated221216
ZeRO 12212/P4 + 12/P
ZeRO 222/P12/P2 + 14/P
ZeRO 32/P2/P12/P16/P

Those are persistent-state formulas. A stage-3 rank still gathers parameters needed by the layer it is about to execute. Prefetching can hide part of that transfer while increasing live memory. Fully Sharded Data Parallel, or FSDP, uses this broad family of parameter gathering and gradient sharding; PyTorch’s fully_shard documentation describes its per-group all-gather and reduce-scatter behavior.

Split the computation itself

Tensor parallelism divides a layer’s matrices. For a two-projection MLP, split the first weight by columns: W₁ = [W₁a W₁b]. Each rank computes its own hidden features φ(XW₁a) or φ(XW₁b). Split the second weight by matching rows. Each rank produces a partial output, then sum those partial outputs. The nonlinearity stays local because it acts independently on hidden coordinates. This column/row pairing is central to Shoeybi et al.’s Megatron-LM.

Pipeline parallelism assigns consecutive layer groups to different devices. A microbatch visits stage 0, then stage 1 and so on; its gradients travel back in reverse. Several microbatches can be in flight at once. The empty startup and drain periods are bubbles, and a slow stage makes every other stage wait.

Synchronous schedule
  • F · forward
  • B · backward
  • Blank · idle
Time slots
—
Idle fraction
—
Peak live microbatches / stage
—

Equal-duration forward and backward tasks, balanced stages, zero communication time. One weight update after all microbatches; basic non-interleaved 1F1B.

Generate schedules from task dependencies. Compare how long forward activations remain live under GPipe and a synchronous 1F1B schedule.

GPipe runs a forward wave followed by a backward wave and accumulates gradients before updating. One-forward-one-backward scheduling, or 1F1B, alternates forward and backward work after warmup so saved activations can be released sooner. The original PipeDream also used weight versions and asynchronous updates. Our visual uses a synchronous, flushed 1F1B schedule with one update after the batch, as distinguished in Narayanan et al. (2021).

For balanced stages with equal forward/backward task durations, zero communication cost, p stages and m microbatches, these basic schedules have idle fraction (p − 1)/(m + p − 1). Basic 1F1B reduces activation lifetimes; it does not magically remove that bubble. Interleaving stage chunks changes the schedule and adds communication.

Combine the dimensions to get 3D parallelism. With 2 data replicas, 4 pipeline stages and 4 tensor-parallel ranks per stage, you use 2 × 4 × 4 = 32 GPUs. Tensor groups cooperate inside layers, pipeline groups exchange activations between layer groups, and data groups combine gradients for corresponding model shards. Sharding state can further reduce redundancy within those data groups. The communication groups follow model ownership, not an arbitrary list of all GPUs.

Check yourselfWhat does basic synchronous 1F1B primarily improve over a forward-then-backward schedule?

Level 08/09Machine

Count the matrix multiplies

So far, the weight update has been a loop, a graph and a distributed protocol. On the GPU it is mostly matrix arithmetic. Open one dense layer and you can derive both the backward work and the familiar training-budget formula.

Flatten batch and token positions into T rows. Let the input X have shape [T, a] and the weight W have shape [a, b]. The forward output is Y = XW, shape [T, b]. Each of its T·b entries is a dot product of length a. Counting a multiply-add as two operations gives approximately 2Tab FLOPs.

Backward arrives with dY = ∂L/∂Y. There are two questions: how should this layer’s weights change, and what error signal should the preceding layer receive? Matrix calculus gives dW = XᵀdY and dX = dYWᵀ. Both have the same leading arithmetic count as forward.

One dense layer becomes three matrix multiplies during training. The transpose changes the contraction direction while preserving the leading FLOP count.

Where 6ND comes from

This layer has N = a·b matrix parameters. Across D token positions, its forward work is about 2ND and its two backward multiplies add about 4ND. Sum over the large dense projections in the model and you get C ≈ 6ND FLOPs. Here N counts the parameters participating in those dominant dense multiplies, and D counts processed training tokens, including tokens seen again in later passes.

The approximation assumes a dense transformer whose parameterized matrix multiplies dominate, a forward/backward pair for each token, and two FLOPs per multiply-add. It omits the optimizer, elementwise operations, communication, and extra recomputation from activation checkpointing. Attention’s score and value products add work that grows with sequence length and is not captured by a parameter count alone. Embedding lookup is also unlike applying every embedding parameter in a dense projection. Using the total published parameter count is therefore a convenient estimate, not an exact kernel inventory.

At long context lengths, or for architectures with sparse expert activation, revise the calculation. A mixture-of-experts model routes each token through only some experts; total parameters and active work are different quantities. Similarly, an input layer that needs no input gradient can skip that backward multiply. Those details explain why counting actual kernels and reporting the chosen FLOP convention matters.

Equal FLOP counts can still take different amounts of time. A matrix multiply with large, well-aligned dimensions gives the device many independent tiles and reuses operands efficiently. Very small batches or narrow shards can leave too little work per kernel. Splitting a model across more GPUs can therefore reduce arithmetic per GPU while also reducing each kernel’s efficiency. Tensor-parallel communication adds another cost. This is why the compute calculator asks for utilization instead of deriving a speedup from GPU count alone.

Choose how much model and how much data

Kaplan et al. (2020) measured smooth empirical relationships between language-model loss, parameters, data and compute. Their compute-optimal allocation favored growing model size faster than training data. Hoffmann et al. (2022), the Chinchilla study, revisited the allocation and found approximately equal scaling of parameters and tokens. Its often-used rule of thumb is about 20 training tokens per parameter in that experimental setting.

Use that rule for a 7B model: D ≈ 140B tokens, so 6 × 7×10⁹ × 140×10⁹ = 5.88×10²¹ FLOPs. Chinchilla itself paired roughly 70B parameters with 1.4T tokens. The ratio is an empirical budget allocation, not a law that a model stops learning after token twenty per parameter. Dataset quality, repeated data, context, architecture and the cost of later inference can change the preferred allocation.

Training tokens D
—
Model FLOPs ≈ 6ND
—
Idealized duration
—
GPU-hours
—

Dense matmul approximation, one multiply-add = 2 FLOPs. Peak is a user-supplied dense figure at the training precision. MFU folds in run overhead; hold it fixed only for rough comparisons.

Turn model size and training tokens into a compute budget, then supply a GPU count, dense peak and utilization to estimate time.

Useful model FLOPs per second divided by the aggregate hardware peak at the relevant training precision., or MFU, connects that work estimate to elapsed time. With a 6ND convention, MFU = (6ND / elapsed seconds) / (GPU count × dense peak FLOP/s). Use the hardware peak for the precision actually being measured, and avoid substituting a sparsity-boosted marketing figure for dense training.

For the 7B example, 64 hypothetical GPUs at 1,000 TFLOP/s each and 40% MFU yield 2.56×10¹⁶ useful model FLOP/s. Dividing the budget by that rate gives about 229,688 seconds, or 2.66 days. This assumes the selected MFU represents the measured interval. If it covers only steady-state steps, add evaluation, checkpointing, failures and setup separately when planning the whole job.

MFU differs from hardware FLOPs utilization when the hardware does extra work such as activation recomputation. Recomputed forward operations occupy tensor cores but do not increase the model’s useful 6ND progress. A memory-saving choice can therefore raise hardware activity while lowering MFU. The right comparison is tokens processed and validation quality per unit of real time and cost.

The same three matrix multiplies explain both backpropagation’s cost and the 6ND estimate. Turning FLOPs into a completion date requires a measured utilization.

Level 09/09Silicon

Move the bytes and survive the run

Matrix kernels only run while their operands are ready. Weights and activations live in GPU high-bandwidth memory, tiles move through on-chip storage, and tensor cores multiply and accumulate those tiles. The inference dive follows that machinery. Training adds two backward multiplies and a network of dependencies between devices, so feeding the arithmetic units becomes a cluster-wide problem.

Two very different distances

Within a supported GPU server or rack-scale domain, NVLink carries GPU-to-GPU traffic and NVSwitch connects larger groups. Between servers, InfiniBand or Ethernet carries traffic through adapters and switches. The exact boundary depends on the platform: an NVLink domain can extend beyond a conventional eight-GPU server. Read the topology before assuming that “same rack” or “same node” identifies a particular link.

Match each training dependency to the fabric it crosses. Frequent tensor-parallel exchanges usually belong on the fastest tightly connected group.

For scale, NVIDIA lists up to 900 GB/s aggregate bidirectional bandwidth per GPU for fourth-generation NVLink and 1,800 GB/s for fifth-generation NVLink on supported systems. These aggregate figures sum multiple links and both directions. They are not the one-way payload rate of one connection or a measured all-reduce throughput.

For inter-server links, InfiniBand NDR and XDR name 400 and 800 Gb/s generations. Dividing by eight gives nominal 50 and 100 GB/s per port before protocol and topology costs. Port counts, routing, oversubscription, competing traffic and collective algorithms determine what a training job sustains. Comparing an aggregate bidirectional GPU number directly with a single network port overstates what those numbers tell you.

Overlap work that has no reason to wait

Backward produces gradients in roughly reverse layer order. Once the gradient bucket for a late layer is complete, data parallelism can start reducing it while the GPU computes gradients for earlier layers. The optimizer must wait for all required reductions, but the network transfer need not wait for the whole backward pass to end.

Bucket size controls a real trade. Small buckets can begin sooner but pay more launch and collective startup overhead. Large buckets use bandwidth efficiently but may become ready too late to hide their transfer. A communication trace with the last collective extending far beyond backward reveals the exposed tail. Measuring that tail is more useful than assuming a nominal fabric rate is the bottleneck.

The same dependency can cross several memory systems. A collective reads a gradient from GPU memory, moves chunks through links and switches, and writes a reduced result back. Meanwhile, backward kernels are reading activations and writing other gradients. Even when the link has spare capacity, memory traffic or launch ordering can keep a transfer from starting promptly. Profiling the ready time, start time and completion time of each bucket separates a slow network from a gradient that simply became available late.

Parameter sharding has a related overlap: gather the next layer’s weights while computing the current layer. Pipeline parallelism can transfer an activation while another stage works on a different microbatch. Each form of overlap consumes resources, and communication can contend with kernels for memory bandwidth or GPU execution capacity. A green utilization bar alone cannot tell you that the overlap saved time.

Make a long run recoverable

A large job also has more opportunities for a GPU, host, link or storage path to fail. Most synchronous jobs cannot simply ignore a missing rank in the middle of a collective. Recovery needs a known checkpoint and a consistent view of which update it contains. A durable checkpoint records model weights, optimizer moments, step and scheduler state, random-number state and sufficient data-loader position to resume the intended data order. FP16 loss-scaler state also belongs there.

Saving only weights is useful for inference or for starting a new optimization run. It does not preserve the trajectory of Adam: its moment estimates depend on past gradients. With state sharding, a checkpoint may be distributed across ranks and must have metadata describing ownership. The system should publish a checkpoint as usable only after all required pieces have been written consistently.

Checkpoint intervals trade write cost against lost work after a failure. A frequent checkpoint interrupts useful processing; an infrequent one can discard hours. Asynchronous writing can reduce pauses if it snapshots consistent tensors and has enough memory and I/O capacity. Evaluate the complete training system, including recovery, rather than measuring only its best uninterrupted minute.

Return to “the cat sat on the”. A token’s probability created a scalar loss. The chain rule sent that loss back through the graph. Adam’s state turned each gradient into a weight change. Memory management kept the needed values alive, collectives made replicas agree, and matrix kernels executed the update on silicon. Repeating that process is where the weights in the inference dive came from.

End of Training · 9 levels, Idea to Silicon

Where to next

Browse all 34 divesAll topics