AI servingDive 01 of 06

How an LLM writes the next word

Follow one prompt through a real model: tokenizer, embeddings, attention, sampling and the KV cache, then down to why serving is bound by memory bandwidth and how tensor cores do the actual work.

Levels
9
Reading
55 min
Depth
Idea to Silicon
Published

Level 01/09Idea

A machine that predicts the next token

Type The cat sat on the into a language model and it carries on: floor, and the cat was still asleep. It looks like the model read your sentence, pictured a cat, and wrote the rest. What actually happened is narrower and stranger. The model computed one list of numbers, a probability for every possible next piece of text, picked one entry from that list, glued it onto your sentence, and did the whole thing again. Nine times.

That is the entire interface. A large language model is a function that takes a sequence of A small unit of text with an integer ID: often a whole word with its leading space, sometimes a word fragment, a punctuation mark or a single byte. and returns a probability distribution over the next token. Chat, code completion, translation, tool calls and "reasoning" are all built on top by formatting text cleverly and calling that function in a loop.

This dive follows that one prompt, The cat sat on the, through a real model from the top to the bottom: how the text becomes numbers, how those numbers flow through attention and the rest of the network, how one token is chosen, and then why the hardware that runs this spends most of its time waiting for memory. The model is GPT-2 small, OpenAI's 124-million-parameter model from 2019. It is small enough to run anywhere, its weights are public, and its skeleton is the same one today's open models use. Every number from GPT-2 on this page (token IDs, embedding values, attention weights, probabilities) came from actually running it on this prompt. Where modern models differ, the text says so.

Why "predict the next token" is such a good interface

It would be reasonable to expect a machine that answers questions to have a question-answering interface. The next-token interface wins for three reasons.

  • Every text is its own training data. Any sentence ever written is a free stack of prediction problems: given the first n tokens, what came next? No labels, no annotators. That is what lets training scale to trillions of tokens.
  • It is complete. A distribution over the next token, applied repeatedly, defines the probability of any whole text: P(text) = P(token 1) × P(token 2 | token 1) × P(token 3 | tokens 1 and 2) × …. Nothing about language is out of reach of this interface in principle.
  • It is one interface for every task. Translation is "here is French, the English is:". A chat is a transcript with the assistant's turn left open. To predict that next token well, the model has to pick up grammar, facts, style, and some ability to follow an argument, because all of those make its guesses better.

Even a chat model sees nothing but a flat token sequence. Here is roughly what a Llama 3 chat model receives when you ask it to finish our sentence: the messages are wrapped in special tokens by a chat template, and the model simply continues the text after the final assistant header.

Llama 3 chat template, as the model sees it
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Finish this sentence: The cat sat on the<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Training made the weights; inference only uses them

Where does the function come from? Training: start with random numbers in every weight matrix, show the network enormous amounts of text, measure how much probability it gave to the token that actually came next, and nudge every weight by gradient descent to make that token a little more likely. GPT-2 was trained this way on about 40 GB of web text. Modern models train on trillions of tokens and then get further fine-tuning to behave like assistants.

This dive is about inference: the weights are frozen, and the only job is to evaluate the function quickly. No learning happens when you chat with a model. Each pass through the network is pure arithmetic on fixed numbers, and the loop around it looks like this.

The·cat·sat·on·the·floor,·and·the·cat·was·still·asleep.
GPT-2 forward passtokens in, one probability for each of 50,257 tokens out
Most likely next token·floor7.6%
Most likely next token,26.1%
Most likely next token·and4.8%
Most likely next token·the12.8%
Most likely next token·cat8.0%
Most likely next token·was8.6%
Most likely next token·still3.5%
Most likely next token·asleep12.4%
Most likely next token.61.7%
Stoppednine tokens generated
  1. The five prompt tokens go through the model in one pass. Out comes a probability for every token in the vocabulary. The single most likely one is ·floor, at only 7.6%.
  2. Pick ·floor (here we always pick the top token, which is called greedy decoding) and append it. The context is now six tokens, and the model runs again. Now a comma is the favourite at 26%.
  3. Append the comma, run again. Each pass produces exactly one new token, so generating a sentence means running the whole network once per token.
  4. The model has no memory between passes except the text itself. Everything it "knows" about the sentence so far is in the token sequence it is handed.
  5. It writes ·cat again. Nothing told it to; repeating the subject is just likely given the words before.
  6. Each step is the same function applied to a slightly longer input.
  7. Probabilities stay low for most picks. Language is genuinely uncertain, and the model's output says so.
  8. Almost done: the clause needs an ending.
  9. A full stop is the favourite, at 62%. Greedy decoding makes the most confident choice it has made so far.
  10. GPT-2 small, greedy, wrote: The cat sat on the floor, and the cat was still asleep. A real system stops at an end-of-sequence token or a length limit. The loop never changes: predict, pick, append, repeat.
1 / 10
The generation loop with GPT-2 small, always taking the most likely token (greedy decoding). The probabilities are the model's own. Step through to watch nine forward passes produce nine tokens.

An LLM is a function from a token sequence to a probability distribution over the next token. Generation is that function in a loop: predict, pick, append, repeat.

Level 02/09Interface

Text becomes tokens

The model never sees letters. Before anything else, a The component that converts text to a list of integer token IDs and back. It is fixed before training and shipped with the model. turns The cat sat on the into five integers:

Two details already matter. The space belongs to the start of the following word (shown here as ·), so ·the (262) and The (464) are different tokens with unrelated IDs. And each of these words happens to be a single token because all five are very common. Rarer words get split.

Why not letters, or whole words?

Letters would make every sequence four to five times longer. That costs twice: attention work grows with the square of the sequence length, and generation produces one token per forward pass, so a character-level model would need several passes per word. Whole words fail the other way. A vocabulary of every word ever written does not exist: there are typos, names, code identifiers, URLs, and a hundred other languages, and any word outside the list becomes unrepresentable.

Subword tokenization sits in between. Frequent words become single tokens; rare words are spelled out from smaller pieces that are themselves frequent. The most common way to build such a vocabulary is byte-pair encoding (BPE). It began as a 1994 data-compression trick by Philip Gage and was adapted for neural machine translation by Sennrich, Haddow and Birch in 2016.

How BPE learns its merges

Training a BPE tokenizer is a short greedy loop, and it is separate from training the model:

  1. Split the training text into chunks with a regular expression (roughly: words with their leading space, runs of digits, runs of punctuation). Merges never cross a chunk boundary.
  2. Write every chunk as its raw UTF-8 bytes. The starting vocabulary is the 256 possible byte values, so any text at all can be represented.
  3. Count every pair of adjacent symbols across the corpus. Merge the most frequent pair into a new symbol, record that merge, and repeat until the vocabulary is the size you want.

To tokenize new text, apply the recorded merges in the order they were learned. The lab below implements exactly this in TypeScript. It is a toy: it trains in your browser on a 445-character paragraph about cats, runs out of repeated pairs after 45 merges, and has nothing to do with GPT-2's real vocabulary. But the algorithm is the real one, and you can watch it work.

0 merges means raw bytes. Vocabulary size is 256 byte tokens plus one token per merge.

The 45 merges this toy learned, in order
  1. 1a+tatid 256, seen 37×
  2. 2h+eheid 257, seen 35×
  3. 3t+hetheid 258, seen 27×
  4. 4·+the·theid 259, seen 24×
  5. 5·+o·oid 260, seen 17×
  6. 6·+s·sid 261, seen 16×
  7. 7·o+n·onid 262, seen 14×
  8. 8·s+at·satid 263, seen 13×
  9. 9·+c·cid 264, seen 12×
  10. 10·c+at·catid 265, seen 12×
  11. 11T+heTheid 266, seen 8×
  12. 12h+athatid 267, seen 5×
  13. 13·+r·rid 268, seen 5×
  14. 14·+The·Theid 269, seen 4×
  15. 15·+a·aid 270, seen 4×
  16. 16·a+n·anid 271, seen 4×
  17. 17·an+d·andid 272, seen 4×
  18. 18·r+at·ratid 273, seen 4×
  19. 19·+m·mid 274, seen 3×
  20. 20·m+at·matid 275, seen 3×
  21. 21·o+the·otheid 276, seen 3×
  22. 22·othe+r·otherid 277, seen 3×
  23. 23·+d·did 278, seen 3×
  24. 24·d+o·doid 279, seen 3×
  25. 25·do+g·dogid 280, seen 3×
  26. 26·+i·iid 281, seen 3×
  27. 27·i+s·isid 282, seen 3×
  28. 28·+hat·hatid 283, seen 2×
  29. 29·+T·Tid 284, seen 2×
  30. 30·T+hat·Thatid 285, seen 2×
  31. 31·the+n·thenid 286, seen 2×
  32. 32·+f·fid 287, seen 2×
  33. 33·f+l·flid 288, seen 2×
  34. 34·fl+o·floid 289, seen 2×
  35. 35·flo+o·flooid 290, seen 2×
  36. 36·floo+r·floorid 291, seen 2×
  37. 37·s+i·siid 292, seen 2×
  38. 38·si+t·sitid 293, seen 2×
  39. 39·sit+t·sittid 294, seen 2×
  40. 40·sitt+i·sittiid 295, seen 2×
  41. 41·sitti+n·sittinid 296, seen 2×
  42. 42·sittin+g·sittingid 297, seen 2×
  43. 43·+b·bid 298, seen 2×
  44. 44·b+e·beid 299, seen 2×
  45. 45·be+d·bedid 300, seen 2×
The training text (445 characters)

The cat sat on the mat. The cat sat on the hat. That cat sat there, and then the cat sat on the other mat. The cats sat on the mats. A rat sat on the cat, and the cat sat on the rat. The rat and the cat sat on the floor. The dog sat on the rug, then the cat sat on the dog. The cat is sitting on the sofa; the dog is sitting on the floor. Then the other cat sat on the bed, and the rat sat on the other bed. The hat sat on the cat. That is that.

A real byte-level BPE, trained live on the small corpus at the bottom. Drag the slider to zero to see raw bytes, then add merges one at a time. Non-English characters fall back to their UTF-8 bytes (shown as <E6> and so on) because this toy corpus never saw them.

Notice that the very first merges are fragments like a + t and h + e, and that whole words such as ·cat only appear after a few rounds. GPT-2's real tokenizer was built the same way on far more text. Its first twelve merges, straight from its merges.txt, are:

1· + t2· + a3h + e4i + n5r + e6o + n7·t + he8e + r9· + s10a + t11· + w12· + o

GPT-2 stopped after 50,000 merges. Its vocabulary is 256 byte tokens, plus 50,000 merged tokens, plus one special <|endoftext|> marker: 50,257 in all. Here is what that vocabulary does to some inputs.

InputGPT-2 tokensIDs
·unbelievable·unbelievable24479
·tokenization·tokenization11241 1634
·2026·20261160 2075
·1234567·123456717031 2231 3134
·Paris·Paris6342
·París·París2547 41200
·ChatGPT·ChatGPT24101 38 11571
·🙂·🙂32485
·東京·<E6><9D><B1><E4><BA><AC>10545 251 109 12859 105
····indent····indent220 220 220 33793

A few lessons hide in that table. ·unbelievable is one token, so the model never directly sees its letters, which is one reason models of this era were bad at spelling and letter-counting tasks. Numbers are chopped into arbitrary chunks (·123, 45, 67), which makes arithmetic harder than it looks; Llama 1 and 2 split every number into single digits for this reason. ·ChatGPT postdates the vocabulary and falls apart into pieces. And the two Japanese characters, six bytes of UTF-8, become five tokens whose boundaries fall in the middle of characters: byte-level BPE never fails, but it can be expensive. The same sentence in a language that was rare in the tokenizer's training data can cost several times as many tokens as in English, and you pay per token in both speed and money.

Modern tokenizers use the same algorithm with bigger vocabularies. Llama 3's has 128,256 tokens, which makes the average token longer and the sequences shorter, at the price of a larger embedding table.

Check yourselfWhy does GPT-2 find it hard to say how many letters are in 'unbelievable'?

Level 03/09Runtime

Tokens become vectors

An integer like 3797 has no useful arithmetic meaning. Token 3798 is not "a bit more cat". The first layer of the network replaces each ID with a vector of real numbers, by looking it up in a learned table: the A matrix with one row per vocabulary entry. Row i is the vector for token i. It is learned during training like every other weight.. In GPT-2 small that table has 50,257 rows of 768 numbers each, 38.6 million parameters, almost a third of the model.

Row 3797 (·cat): first 16 of 768 numbers
  1. 00.010
  2. 10.037
  3. 20.164
  4. 3-0.218
  5. 40.029
  6. 5-0.098
  7. 6-0.336
  8. 7-0.075
  9. 80.113
  10. 9-0.095
  11. 10-0.036
  12. 11-0.197
  13. 12-0.018
  14. 130.034
  15. 140.098
  16. 15-0.175

Length of the full 768-number vector: 3.258. These are the actual trained values in GPT-2 small.

Closest rows to ·cat by cosine similarity
  1. ·Cat0.71
  2. ·cats0.69
  3. Cat0.64
  4. cat0.60
  5. ·dog0.55
  6. ·Cats0.54
  7. ·CAT0.53
  8. ·kitten0.50
  9. cats0.49
  10. ·kittens0.47
Token 3797 selects one row of GPT-2's embedding matrix. Middle: the first 16 of its 768 values, exactly as stored. Right: the rows most similar to it, measured by the cosine of the angle between vectors.

Nobody chose those numbers. Training moved them wherever they helped prediction, and similar tokens ended up pointing in similar directions: ·cat sits near ·Cat, ·cats, ·dog and ·kitten. The individual coordinates do not have human-readable meanings. The geometry is what carries information.

From here on, each of our five tokens is a vector of 768 numbers, and the network's job is to keep transforming those vectors until the one at the last position, ·the, contains enough information to predict what comes after it. This running vector per position is called the The vector each position carries from layer to layer. Every attention and MLP block reads it, computes something, and adds its result back onto it., and every later layer reads from it and adds to it.

The network needs to know the order

There is a gap. The embedding of ·cat is the same wherever the word appears, and attention by itself has no notion of order (the causal mask leaks some position information, but models learn far better with an explicit signal). Without extra information, The cat sat on the and the on sat cat The would look identical. The model needs to know positions.

GPT-2 handles this the simple way: a second learned table with one 768-number vector per position (1,024 of them), added to each token's embedding. It works, but it fixes the maximum context at 1,024 tokens and teaches the model absolute positions, when what usually matters is relative distance: "the word right before me", "the subject three words back".

Most current open models (the Llama, Mistral and Qwen families, among others) take a different route, rotary position embedding (RoPE), which works inside attention itself. It gets its full explanation in the next level, right after queries and keys are introduced.

Embeddings turn token IDs into learned vectors. Position comes either from an added position vector (GPT-2) or, in most newer models, from a rotation applied inside attention (RoPE).

Level 04/09Runtime

Attention and the transformer block

Right now the vector at the last position knows two things: "I am ·the" and "I am at position 4". To predict ·floor, it needs information that lives at other positions: there is a cat, it sat, the cat sat on something. Attention is the only part of the network that moves information between positions. Everything else works on each position separately.

Queries, keys and values

Each position computes three vectors from its residual-stream vector, using three learned weight matrices:

  • a query, q: what this position is looking for;
  • a key, k: what this position offers to anyone looking;
  • a value, v: what this position will hand over if it is picked.

Position i scores every position j by the dot product qi·kj, divided by √d (d is the length of each vector). A Setting the score to minus infinity for every key that comes after the query, so softmax gives it exactly zero weight. removes every j later than i, because when generating, the future does not exist yet. Softmax turns each row of scores into weights that are positive and sum to 1, and the output is the weighted average of the values. In matrix form, from the 2017 paper that introduced the transformer:

Attention(Q, K, V) = softmax(QKᵀ / √d + mask) · V

one attention head in NumPy
import numpy as np

def attention(x, Wq, Wk, Wv):
    q, k, v = x @ Wq, x @ Wk, x @ Wv          # (T, d) each
    scores = q @ k.T / np.sqrt(q.shape[-1])   # (T, T)
    T = len(x)
    scores[np.triu_indices(T, 1)] = -np.inf   # causal mask: no peeking ahead
    w = np.exp(scores - scores.max(-1, keepdims=True))
    w /= w.sum(-1, keepdims=True)             # softmax, row by row
    return w @ v                              # (T, d): a weighted mix of values

The division by √d exists because a dot product of two vectors with d random components has a spread that grows like √d. Without it, big heads would produce huge scores, and softmax would saturate into a hard one-hot choice with almost no gradient during training.

Where position comes in: RoPE

Most current open models (the Llama, Mistral and Qwen families, among others) use rotary position embedding, RoPE, introduced by Su and colleagues in 2021, instead of GPT-2's added position vectors. RoPE adds nothing to the residual stream. Instead, just before the q·k dot products are taken, it rotates every query and key by an angle proportional to its position. In Llama-style models each head's queries and keys have 128 numbers:

  • Split each query or key into 64 pairs, and treat each pair as a point in a plane.
  • Rotate pair j at position m by the angle m·θj, where θj = base−2j/128. Pair 0 turns a full radian per position; the last pairs turn almost not at all.
  • When a query at position m meets a key at position n, the rotations partly cancel. The dot product depends only on the offset n − m, never on m or n alone.
Which pair of dimensions (head size 128, base 10,000)

query rotated by m·θ =

key rotated by n·θ =

offset n − m =

dot product q·k for this pair =

Dot product against offset n − m (only the offset matters)
−400+40
RoPE for one pair of dimensions. Move the two positions and the vectors rotate. Press a shift button: both positions move by the same amount, both vectors rotate further, and the dot product does not change. Switch to a slower pair to see how low frequencies change little over short distances.

The different frequencies work like the hands of a clock. Fast-turning pairs distinguish neighbours, slow pairs keep track of distances in the thousands. Llama 2 uses base 10,000; Llama 3 raised it to 500,000 so the slowest pairs turn even more slowly, which helps with longer contexts.

Below is a complete attention computation, running in your browser, on our five tokens. To make it readable the inputs are hand-made six-number vectors with named features, and the two heads have hand-set weights. A trained model learns its weights, and its features are not labelled. But every number shown is computed, not drawn: change a setting and the matrices are recomputed.

The input vectors (one row per token, hand-made features)
tokenconstthingactionrelationdetermineranimal
The100010
·cat110001
·sat101000
·on100100
·the100010

Scores: q · k / √d

Weights after softmax (pick a cell)

A live, toy attention layer with two heads of size 4 and RoPE. Head 1 ignores content and uses only position: its key is the query rotated one step, so the best match is always the previous token. Head 2 ignores position: at a determiner, it looks for a thing. Pick a cell to see its dot product, and try switching RoPE or the mask off.

Spend a minute with the toggles. Turn RoPE off and Head 1 goes flat: every score in a row is identical, because without position information a content-blind head cannot tell its neighbours apart. Turn the mask off and earlier positions start reading from later ones, which would be cheating during training and impossible during generation. Turn off the √d scaling and the softmax gets sharper.

Then look at what position 4, ·the, gets back. Head 1 returns "the previous token was a relation word" (it read ·on), Head 2 returns "there is an animal thing" (it read ·cat). Added back onto the residual stream, that is exactly the kind of information that makes ·floor a good guess.

Many heads, and what real ones do

One head computes one weighted average, so it can only look for one thing at a time. Transformers run several heads side by side on smaller slices: GPT-2 small has 12 heads of 64 dimensions in each of its 12 layers. Their outputs are concatenated back to 768 numbers and mixed by one more matrix, WO.

Here are the real attention patterns of all 144 heads in GPT-2, recorded while it processed the sentence it wrote earlier. The presets are heads with well-known behaviour; the selectors let you browse the rest.

Pick a query token in the sentence. Earlier tokens light up by how much attention it pays them.

Full pattern for this head: row = query, column = key, stronger colour = more weight. The upper triangle is empty because of the causal mask.

Real attention weights from GPT-2 small on its own greedy output. Head 4.11 (layer 4, head 11, counting from zero) is a near-perfect previous-token head. Head 5.5 is an induction head: at the second ·cat it attends to ·sat, the token that followed ·cat last time. Head 7.2 sends almost everything to the first token.

The previous-token head (4.11) is the trained version of our toy Head 1, learned without RoPE through GPT-2's position embeddings. The induction head (5.5) is a famous circuit, described in detail by Olsson and colleagues in 2022: it finds earlier occurrences of the current token and looks at what came next, which is a basic mechanism for copying patterns from the context. The duplicate-token head (0.1) attends to earlier copies of the same word.

And then there is head 7.2, which ignores the sentence and stares at The. Softmax forces every row to sum to 1, so a head that has nothing useful to do still has to put its attention somewhere, and many heads learn to park it on the first token. Averaged over all 144 heads and every position after the first, 61% of GPT-2's attention on this sentence goes to position 0. Xiao and colleagues named this an attention sink in 2023, and it matters in practice: if a serving system evicts the first tokens from a long context, quality can collapse.

The rest of the block: MLP, residuals and normalization

Attention moves information between positions. The MLP (multi-layer perceptron) then transforms each position on its own: in GPT-2, expand 768 numbers to 3,072, apply the GELU nonlinearity, and project back down to 768. The MLPs hold two thirds of each block's parameters, and interpretability work suggests that much of a model's stored factual association lives in them. Llama-style models use a gated variant, SwiGLU, with three matrices instead of two.

Each sub-block reads a normalized copy of the residual stream and adds its output back rather than replacing it. Those additions are the residual connections, and they are why the stream is called that: each layer writes a small edit onto a shared running vector. GPT-2 uses LayerNorm (rescale each vector to zero mean and unit variance, then apply a learned scale and shift) placed at the input of each sub-block. Llama uses the cheaper RMSNorm, which skips the mean.

one transformer block, schematically
def block(x, p):
    x = x + attention(layer_norm(x, p.ln1), p.attn)   # mix information across positions
    x = x + mlp(layer_norm(x, p.ln2), p.mlp)          # transform each position on its own
    return x

def mlp(x, p):
    return gelu(x @ p.W_up + p.b_up) @ p.W_down + p.b_down   # 768 -> 3072 -> 768
× 12 blockstoken + position vectorsresidual stream · 768 per tokenLayerNormAttention12 heads × 64 dims+LayerNormMLP768 → 3,072 → 768, GELU+final LayerNormUnembedding768 → 50,257 logitssame matrix as theembedding, transposed

Where GPT-2 small's 124.4 million parameters live

PartShapeParameters
Token embedding (shared with unembedding)50,257 × 76838,597,376
Position embedding1,024 × 768786,432
Attention, per block: Q, K, V and output768 × 2,304 + 768 × 7682,362,368
MLP, per block768 × 3,072 + 3,072 × 7684,722,432
LayerNorms, per block4 × 7683,072
All 12 blocks12 × 7,087,87285,054,464
Final LayerNorm2 × 7681,536
Total124,439,808

Counts include biases. Attention and MLP rows include their bias vectors (2,304 + 768 and 3,072 + 768).

A GPT-2 block and where the parameters are. Twelve of these are stacked. After the last, a final LayerNorm and the unembedding (the embedding matrix used in reverse) turn the vector at each position into 50,257 scores.

Attention is the only step where positions exchange information; the MLP then processes each position alone. Both add their results onto a residual stream that flows through every layer.

Check yourselfIn the toy above, why does Head 1 spread its attention evenly when RoPE is off?

Level 05/09Runtime

From logits to one token

After 12 blocks, the vector at the last position goes through a final LayerNorm and is multiplied by the unembedding matrix, 768 × 50,257. In GPT-2 that matrix is the embedding table, transposed: the output vector is scored against every token's embedding. The result is one number per vocabulary entry, a An unnormalized score. Softmax turns a list of logits into probabilities; only differences between logits matter.. For our prompt, the top three are ·floor at −80.65, ·bed at −80.81 and ·couch at −81.00. GPT-2's logits happen to sit around −80; softmax only cares about differences, so the offset is meaningless.

Softmax with a temperature T turns logits z into probabilities:

pi = exp(zi / T) / Σj exp(zj / T)

At T = 1 this is the model's own distribution, and it is remarkably flat. The favourite, ·floor, gets 7.6%. The "obvious" answer ·mat is ranked 31 with 0.4%. The entropy is about 7.8 bits, the same uncertainty as a fair choice among about 225 tokens, and the top 200 tokens together hold only 79% of the probability: the other fifth is spread over 50,000 unlikely options.

Turning that distribution into one token is the job of the sampler, a few lines of ordinary code outside the network. It has three classic knobs:

  • Temperature divides the logits before softmax. Below 1 it sharpens the distribution toward the favourites; above 1 it flattens it. At 0 it becomes greedy: always the single top token.
  • Top-k keeps only the k most likely tokens and renormalizes (Fan, Lewis and Dauphin, 2018).
  • Top-p, or nucleus sampling, keeps the smallest set of top tokens whose probabilities add up to at least p (Holtzman and colleagues, 2019). Unlike top-k it adapts: few tokens when the model is confident, many when it is not.
  1. ·floor
  2. ·bed
  3. ·couch
  4. ·ground
  5. ·edge
  6. ·bench
  7. ·table
  8. ·sofa
  9. ·other
  10. ·back
  11. ·side
  12. ·chair
  13. ·stool
  14. ·top
  15. ·mat rank 31
  16. the other 50,242

Outline: the model's raw distribution at temperature 1. Fill: what the sampler actually draws from after your settings. The last bar is drawn on a 0 to 100% scale.

The prompt is The cat sat on the. Sampling picks the next token.

A sampling playground over GPT-2's real next-token distribution for our prompt. The top 200 logits are stored exactly; the other 50,057 are stored as a histogram, which is accurate enough for every number shown. Try temperature 0.3, then top-p 0.5, then temperature 1.8.
temperature, top-k and top-p, in the order Hugging Face transformers applies them
def sample(logits, temperature=1.0, top_k=0, top_p=1.0):
    if temperature == 0:
        return int(np.argmax(logits))                  # greedy
    z = logits / temperature
    order = np.argsort(-z)
    if top_k > 0:
        z[order[top_k:]] = -np.inf                     # keep the k best
    p = np.exp(z - z.max()); p /= p.sum()
    if top_p < 1.0:
        cum = np.cumsum(p[order])
        cut = np.searchsorted(cum, top_p) + 1          # smallest set reaching top_p
        p[order[cut:]] = 0; p /= p.sum()
    return int(np.random.choice(len(p), p=p))

Why not always take the top token? Greedy decoding is deterministic and fine for short factual answers, but over long passages it tends to fall into loops and bland repetition: the most likely token at each step does not add up to the most likely or most natural text. Sampling with a moderate temperature and a nucleus cutoff keeps text varied while cutting off the long tail of nonsense. The order of the filters matters, and libraries differ, so the same settings can give slightly different distributions in different servers.

Because the sampler only edits logits, it is also where structured output happens. To force valid JSON, a server can set the logit of every token that would break the grammar to −∞ before sampling. The network never knows.

The network ends in a vector of logits. Everything about how "creative" the output is lives in a small, separate sampling step that you can change without touching the model.

Level 06/09Runtime

Prefill, decode and the KV cache

The loop in the first level reran the model on the whole sequence for every new token. That is correct, and very wasteful. Look again at the causal mask: a token's keys and values depend only on itself and the tokens before it. When ·floor is appended, the keys and values already computed for The cat sat on the do not change in any layer. So keep them.

That is the KV cache, and it splits inference into two phases with very different characters:

  • Prefill processes the whole prompt at once. All five tokens go through every layer together, each weight matrix multiplies a 5 × 768 block of activations, and every layer's keys and values are written to the cache. The last position's output gives the first new token. The time this takes is the time to first token.
  • Decode then produces one token per step. Only the newest token goes through the network. Its query attends over all the cached keys, and its own key and value are appended. The time per step is the inter-token latency, the speed at which text streams out.
prefill once, then decode token by token (sketch)
cache = KVCache(layers=12)
logits = model(prompt_ids, cache)          # prefill: all prompt tokens at once
for _ in range(max_new_tokens):
    next_id = sample(logits[-1])
    if next_id == EOS:
        break
    logits = model([next_id], cache)       # decode: one token, attends over the cache
tokens
The·cat·sat·on·the·floor,·and·the·cat·was·still·asleep.
keyvaluewritten this step12 layers top to bottom
phase
tokens pushed through the weights
attention scores computed
cache size, FP16
total token passes, with cache / without
  1. Prefill. All five prompt tokens go through the network together, as one batch of matrix multiplies. Every layer computes a key and a value vector for every token and writes them to the cache. The last position's output becomes the first prediction: ·floor.
  2. Decode, step 1. Only the new token, ·floor, goes through the network. Its query attends over the six cached keys (five old plus its own), and its own K and V are appended. One token's worth of matrix work instead of six.
  3. Each decode step adds exactly one column to the cache in every layer. Nothing already in the cache is ever recomputed: a token's keys and values depend only on the tokens before it, and those never change.
  4. The work per decode step is tiny, one token, but each step needs the whole model's weights. That mismatch is the subject of the next level.
  5. The attention part of each step grows with the context: the new query is compared with every cached key. At 14 tokens this is nothing; at 100,000 tokens it is a lot of memory to read.
  6. Without a cache, step n would push all n tokens through again. The bottom meter keeps the running total both ways.
  7. Cache size grows linearly: 36 KiB per token for GPT-2 small in FP16.
  8. For big models the per-token figure is far larger (the serving level does the arithmetic), and a long conversation's cache can outweigh a small model.
  9. Thirteen tokens cached. The model has made 13 token passes in total, against 81 if every step recomputed the whole sequence. The gap grows with the square of the length.
  10. A tenth pass the loop in the first level did not need: generation already chose the full stop, so this pass only pushes it through to cache its keys and values (its prediction is thrown away). That makes 14 passes instead of 95. If the conversation continues, a server can keep this cache so the next message only needs to prefill its own new tokens.
1 / 10
Prefill and decode on GPT-2 with a KV cache. Each column of the grid is one token's keys (blue) and values (amber) in each of the 12 layers. Decode steps add one column. The meters compare the work with and without the cache.

How big the cache gets

Per token, the cache stores one key and one value vector per layer:

KV bytes per token = 2 × layers × KV heads × head size × bytes per number

For GPT-2 small in 16-bit floats: 2 × 12 × 12 × 64 × 2 = 36 KiB per token, so a full 1,024-token context is 36 MiB. Bigger models change the picture quickly. One design choice matters a lot here: in grouped-query attention (Ainslie and colleagues, 2023), several query heads share one key/value head, so only the KV heads need caching. The extreme version, one KV head for all queries, is multi-query attention (Shazeer, 2019).

ModelLayersQuery heads / KV headsKV cache per token (FP16)One 4,096-token sequence
GPT-2 small1212 / 1236 KiB144 MiB (if it could)
Llama 2 7B3232 / 32512 KiB2 GiB
Llama 3 8B3232 / 8128 KiB512 MiB
Llama 3 70B8064 / 8320 KiB1.25 GiB

A Llama 2 7B model weighs 13.5 GB in FP16, and just seven 4,096-token conversations add another 14 GiB of cache. That pressure is why newer models use grouped-query attention: Llama 3 70B has 64 query heads but only 8 KV heads, an 8× smaller cache than full multi-head attention would need, at a small cost in quality.

Servers also reuse caches across requests. If thousands of requests start with the same long system prompt, its keys and values can be computed once and shared, which is called prefix caching.

Causal attention means past keys and values never change, so they are cached. Prefill processes the prompt in parallel; decode then pushes one token at a time through the network.

Level 07/09System

Serving: bound by memory bandwidth

Now count what one decode step costs. The newest token must be multiplied by every weight matrix in the model. Each weight is used for exactly one multiply and one add, two floating-point operations, and to do that the weight has to travel from the GPU's main memory to its arithmetic units.

Take Llama 2 7B in 16-bit precision on one NVIDIA H100 SXM. NVIDIA's datasheet gives it 3.35 TB/s of HBM3 memory bandwidth and about 989 TFLOPS of dense FP16 tensor throughput (the often-quoted 1,979 assumes structured sparsity).

  • Memory: about 6.6 billion weights × 2 bytes = 13.2 GB to read per step. At 3.35 TB/s that takes at least 3.9 ms.
  • Compute: 2 × 6.6 billion = 13.2 GFLOP. At 989 TFLOPS that takes 13 µs.

Memory is about 300 times slower than math. At batch size 1 the arithmetic units sit idle more than 99% of the time, and the speed limit, about 250 tokens per second (a little less once the KV cache is counted), comes entirely from memory bandwidth. This is the single most important fact about serving LLMs.

Arithmetic intensity and the roofline

The tool for reasoning about this is the roofline model (Williams, Waterman and Patterson, 2009). A computation's FLOPs performed per byte moved to or from main memory. High intensity means data is reused many times once loaded. is its FLOPs divided by the bytes it moves. A chip's ridge point is its peak FLOPS divided by its bandwidth: for the H100 in FP16, 989 / 3.35 ≈ 295 FLOPs per byte. Below the ridge, a workload is limited by bandwidth; above it, by compute. Decode at batch 1 in FP16 does 2 FLOPs per 2-byte weight: an intensity of 1. It is about as far left on the roofline as a workload can be.

The way out is batching. If 64 users are decoding at once, each weight read from memory is used 64 times, once per sequence. Weight traffic stays the same, FLOPs go up 64×, and total throughput rises almost for free. But there is a catch that the calculator makes visible: every sequence has its own KV cache, and attention has to read all of it every step. That traffic grows with batch size and context length and is never shared. For Llama 2 7B at batch 64 with 1,024 tokens of context each, the caches (34 GB) already outweigh the model (13 GB), and the step is still memory-bound.

arithmetic intensity, FLOPs per byte (log)attainable TFLOPS (log)

Horizontal: arithmetic intensity, FLOPs per byte (log). Vertical: attainable TFLOPS (log).

Bytes read per step
FLOPs per step
Intensity / ridge point
Step time bound (memory / compute)
Tokens per second, each sequence
Tokens per second, whole batch
KV cache per token / for the batch
Memory needed / available
Prefill of one context-length prompt, at least

Upper bounds from the datasheet peaks (dense, no sparsity). Real kernels reach a fraction of peak bandwidth and compute, and splitting across GPUs costs communication this ignores. FP8 compute is used only for FP8 weights on GPUs that support it; INT4 weights are assumed to be widened to FP16 for the math.

A roofline calculator for one decode step. It counts the bytes of weights and KV cache that must be read and the FLOPs that must be done, and divides by the GPU's datasheet bandwidth and peak. Start at batch 1, then raise the batch, then raise the context.

Prefill sits at the other end. A 1,000-token prompt uses every weight 1,000 times per read, so its intensity is about 1,000 FLOPs per byte, comfortably compute-bound. Prefill and decode are such different workloads that some large deployments run them on separate pools of GPUs.

Continuous batching

Real requests arrive at random times and want different numbers of tokens. The naive approach, gather a batch, run it until every request is done, then start the next, leaves slots idle whenever a short answer finishes early. Orca (Yu and colleagues, 2022) introduced iteration-level scheduling, now usually called continuous batching: after every forward pass, finished sequences leave the batch and waiting ones join.

Static batching19 iterations to finish all eight requests, slots busy 62% of the time

Continuous batching13 iterations to finish all eight requests, slots busy 90% of the time

Eight requests (A to H) with different output lengths share four batch slots. One column is one forward pass of the model. A lettered cell is the request's prefill; hatched cells are slots doing nothing.

The same eight requests scheduled two ways on a server with four batch slots. Simplified: prefill is shown as one iteration, and real servers also split long prefills into chunks.

Paged attention

Continuous batching makes memory management hard. Sequences grow one token at a time and nobody knows in advance how long they will get. Reserving a contiguous region for each sequence's maximum length wastes most of it; the vLLM paper (Kwon and colleagues, 2023) measured existing systems using only 20 to 40% of their KV-cache memory for actual tokens. PagedAttention borrows the operating system's idea of virtual memory: split the cache into fixed-size blocks (16 tokens by default in vLLM), allocate them on demand, and keep a per-sequence block table that maps logical blocks to physical ones. Only the last block of each sequence is ever partly empty, and identical prefixes can share physical blocks.

Sequence A: 41 tokens

logical012
physical713

Sequence B: 20 tokens

logical01
physical49

Sequence C: 9 tokens

logical0
physical0
A paged KV cache. Each sequence's block table maps its logical blocks to arbitrary physical blocks, so memory is allocated 16 tokens at a time and never needs to be contiguous. The attention kernel follows the table.

Quantization: fewer bytes per weight

If decode time is bytes divided by bandwidth, the most direct fix is fewer bytes. Storing weights in 8-bit floating point (supported natively by H100 tensor cores) halves the bytes per step and nearly doubles the batch-1 speed limit. 4-bit integer weights, produced by methods like GPTQ (Frantar and colleagues, 2022) and AWQ (Lin and colleagues, 2023), cut it by four, at the cost of unpacking them to 16 bits inside the kernel and some loss of accuracy that has to be measured per model. The KV cache can be quantized the same way. Try FP8 and INT4 in the calculator: the decode bound moves almost exactly in proportion.

Speculative decoding: guess, then check in parallel

Memory-bound decode has a strange property: running the big model on 5 tokens costs about the same as running it on 1, because the weights are read once either way. Speculative decoding (Leviathan, Kalman and Matias; and Chen and colleagues, both 2023) exploits that. A small, fast draft model guesses the next γ tokens. The big model then scores all of them in a single forward pass and accepts the guesses left to right using a rejection-sampling rule, then adds one token of its own. The rule is constructed so that the output follows exactly the big model's distribution (with greedy decoding, exactly the same text), only sooner.

Bar i: the chance that one round produces at least i tokens. The first is always 100%, because on a rejection the target model supplies its own replacement token.

Expected gain from speculative decoding, using the formula from Leviathan and colleagues. It assumes each draft token is accepted independently with probability α and that verifying γ + 1 tokens costs one ordinary decode step, which is close to true when decode is memory-bound.
TechniqueWhat it attacksWhat it costs
KV cacheRecomputing the past every stepMemory that grows with batch × context
Batching, continuous batchingWeight reads, amortized across sequencesEach sequence waits for the whole batch's step
Grouped-query attentionKV-cache size and readsDecided at training time; small quality cost
PagedAttentionWasted, fragmented cache memoryIndirection in the attention kernel
QuantizationBytes per weight (and per cached value)Accuracy, kernel complexity
Speculative decodingNumber of sequential big-model stepsA draft model, wasted work on rejected guesses

Decode reads every weight to produce one token per sequence, so it is bound by memory bandwidth. Nearly every serving technique is a way to move fewer bytes per token or to share each byte across more tokens.

Check yourselfLlama 2 7B in FP16 on one H100, 1,024 tokens of context each. You go from batch 8 to batch 16. Roughly what happens?

Level 08/09Machine

Everything is a matrix multiply

Strip away the names and a transformer step is a short list of matrix multiplications with a few cheap elementwise operations (normalization, GELU or SiLU, softmax, the RoPE rotation) between them. Here is one Llama 2 7B layer during decode, with B sequences in the batch:

OperationShapesFLOPs per token
Q, K, V projections (fused)[B × 4096] · [4096 × 12288]100.7 M
Attention scores, q · Kᵀ32 heads × [1 × 128] · [128 × context]8,192 × context
Weighted values, weights · V32 heads × [1 × context] · [context × 128]8,192 × context
Output projection[B × 4096] · [4096 × 4096]33.6 M
MLP gate and up[B × 4096] · [4096 × 22016]180.4 M
MLP down[B × 11008] · [11008 × 4096]90.2 M

A multiply of an [M × K] matrix by a [K × N] matrix does M·N·K multiply-adds, which is 2·M·N·K FLOPs. With one token per sequence, M is just the batch size, and each weight matrix contributes 2 FLOPs per weight per token. Add up all 32 layers plus the final projection onto the 32,000-token vocabulary and you get the rule of thumb:

FLOPs per generated token ≈ 2 × parameters (+ 4 × layers × dmodel × context for attention)

For Llama 2 7B that is about 13.2 GFLOP per token, plus about 2.1 GFLOP more for attention at a 4,096-token context. For GPT-2 small, about 0.25 GFLOP. (Training costs roughly 6 × parameters per token, because the backward pass does about twice the work of the forward pass, which is where the weights in this dive came from at enormous expense.)

With M = 1, a matrix multiply degenerates into a matrix-vector product, and every weight is touched once and thrown away. That is the decode problem from the previous level, restated in linear-algebra terms. With M = 1,000 during prefill, each weight is reused 1,000 times.

Tiling: how a GPU actually multiplies matrices

Reuse is also the central problem inside a single matrix multiply. Computing one element of C = A × B needs a full row of A and a full column of B. Done naively, every multiply-add fetches two numbers from main memory, and the chip starves.

The fix is tiling. Split C into tiles and give each tile to a group of threads. That group walks along the shared dimension K in chunks: it loads a tile of A and a tile of B into fast on-chip memory, does every multiply-add those two tiles allow, and moves on. Each number loaded is reused once for every row or column of the tile, so bigger tiles mean fewer trips to main memory.

Tile size

A (8 × 8)

B (8 × 8)

C = A × B

step
elements loaded from global memory
multiply-adds done
FMAs per element loaded
A tiled 8 × 8 matrix multiply with real integer values. Each step loads one tile of A (blue) and one tile of B (amber) and adds their product into one tile of C. Compare the global-memory loads for tile sizes 1, 2, 4 and 8: the total is 2·8·8·8 divided by the tile size.

On a real GPU the tile of C is owned by a thread block, the tiles of A and B are staged in shared memory, and the tile is limited by how much shared memory and how many registers the block can use. Here is the textbook version in CUDA. Production libraries such as cuBLAS and CUTLASS add a second level of tiling in registers, asynchronous copies so loading overlaps computing, and, above all, tensor cores.

matmul_tiled.cu: one thread per output element, 16 × 16 tiles
#define T 16

__global__ void matmul_tiled(const float* A, const float* B, float* C, int M, int N, int K) {
    __shared__ float As[T][T];
    __shared__ float Bs[T][T];
    int row = blockIdx.y * T + threadIdx.y;
    int col = blockIdx.x * T + threadIdx.x;
    float acc = 0.0f;
    for (int k0 = 0; k0 < K; k0 += T) {
        As[threadIdx.y][threadIdx.x] = (row < M && k0 + threadIdx.x < K) ? A[row * K + k0 + threadIdx.x] : 0.0f;
        Bs[threadIdx.y][threadIdx.x] = (k0 + threadIdx.y < K && col < N) ? B[(k0 + threadIdx.y) * N + col] : 0.0f;
        __syncthreads();
        for (int k = 0; k < T; k++)
            acc += As[threadIdx.y][k] * Bs[k][threadIdx.x];
        __syncthreads();
    }
    if (row < M && col < N)
        C[row * N + col] = acc;
}

Inference is a sequence of matrix multiplies costing about 2 FLOPs per parameter per token. Whether they run fast depends on reuse: how many times each number fetched from memory is used before it is thrown away.

Check yourselfIn the tiled multiply, what does making the tiles larger change?

Level 09/09Silicon

Tensor cores, HBM and FlashAttention

At the bottom of the stack is a chip doing those tiles. Take the H100 SXM, as described in NVIDIA's Hopper architecture whitepaper and datasheet. It has 132 streaming multiprocessors (SMs), the GPU's cores. Each SM is split into four sub-partitions, each with its own scheduler, its own slice of the register file and its own tensor core, and all four share a 256 KB block of fast SRAM that serves as L1 cache and programmer-managed shared memory.

H100 SXM at two scales, simplified from NVIDIA's Hopper whitepaper. Left: 132 SMs sharing a 50 MB L2 cache, next to five HBM3 stacks. Right: one SM, the unit that runs a thread block and holds its tiles.

Warps

GPU threads are scheduled in groups of 32 called A group of 32 GPU threads that execute the same instruction at the same time, each on its own data.. A warp issues one instruction at a time for all 32 threads together. An SM keeps many warps resident at once, and each clock each scheduler picks one that is ready to go. When a warp stalls waiting for memory, which takes hundreds of cycles, another runs instead. GPUs hide memory latency with sheer parallelism rather than with large caches.

Tensor cores

The ordinary FP32 units can do one fused multiply-add per thread per clock. Tensor cores are specialized hardware that performs a whole small matrix multiply-accumulate, D = A × B + C, as one operation. In PTX, NVIDIA's virtual instruction set, a warp-wide tensor core instruction looks like this:

a warp-level matrix multiply-accumulate in PTX (sm_80 and later)
// One warp (32 threads) computes D = A x B + C for a 16x8 tile, k = 16.
// A: 16x16 FP16, 4 registers per thread. B: 16x8 FP16, 2 registers.
// C and D: 16x8 FP32, 4 registers per thread.
mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32
    {%f0, %f1, %f2, %f3},
    {%r0, %r1, %r2, %r3},
    {%r4, %r5},
    {%f4, %f5, %f6, %f7};

That single instruction is 16 × 8 × 16 = 2,048 multiply-adds, with the operands spread across the 32 threads' registers. Clock for clock, an A100 SM does 1,024 dense FP16 multiply-adds; Hopper doubles that to 2,048 per SM, which at about 1.83 GHz across 132 SMs works out to the datasheet's 989 TFLOPS. The same chip's non-tensor peak is about 67 TFLOPS in FP32 and about 134 in FP16 (NVIDIA quotes those at the 1.98 GHz boost clock, while the 989 TFLOPS tensor figure corresponds to about 1.83 GHz). Matrix multiplies that miss the tensor cores run roughly 7 to 15 times slower. Hopper also adds warpgroup instructions (wgmma), where four warps together issue larger asynchronous multiplies that read their operands straight from shared memory, and a Tensor Memory Accelerator (TMA) that copies whole tiles from global memory into shared memory without tying up threads.

The memory hierarchy is the real story

Tensor cores are so fast that the question is never "can it compute this?" but "can it get the numbers there in time?" Every level of the hierarchy is a trade between size and speed:

Level (H100 SXM)SizeWhere it isNotes
Registers256 KB per SM, ~33 MB totalInside each SM sub-partitionOperands of tensor core instructions live here
Shared memory / L1256 KB per SM (up to 228 KB shared)Inside each SMWhere tiles are staged; managed by the kernel
L2 cache50 MBOn the die, shared by all SMsHardware-managed
HBM380 GBFive DRAM stacks beside the dieAbout 3.35 TB/s; holds weights and KV cache

NVIDIA does not publish a single on-chip SRAM bandwidth figure for H100. The FlashAttention paper gives a useful order of magnitude for the A100: about 19 TB/s for SRAM against 1.5 to 2.0 TB/s for HBM, roughly ten times faster, but over a few tens of megabytes instead of tens of gigabytes. The physics behind the gap is that HBM is DRAM: each bit is a tiny capacitor read through a long, wide bus (1,024 bits per stack, 5,120 bits for five stacks), dense and relatively slow. SRAM spends six transistors per bit to be fast, and so can only be small. Moving a number in from DRAM also costs far more energy than doing arithmetic on it; Mark Horowitz's widely cited 45 nm figures put a 32-bit DRAM read at roughly 640 pJ against about 4 pJ for a 32-bit floating-point multiply.

FlashAttention: tiling attention so it fits on chip

Standard attention computes the full score matrix S = QKᵀ, writes it to HBM, reads it back for softmax, writes the probabilities, and reads them again to multiply by V. For a long sequence that matrix is enormous (a 32,000-token context gives a billion scores per head) and the traffic, not the arithmetic, dominates. FlashAttention (Dao and colleagues, 2022) never writes it. It loads a block of queries into SRAM, streams blocks of keys and values past it, and keeps a running result. The obstacle is softmax, which seems to need the whole row before it can normalize. The fix is the online softmax (Milakov and Gimelshein, 2018): keep a running maximum and a running sum, and rescale what you have accumulated whenever the maximum changes.

score s1.2-0.52.80.33.1-10.72.2value v41-23502-1
afterrunning max mmax mrunning sum ℓ = Σ es−msum ℓaccumulator Σ es−m·voutput so far, acc / ℓoutput so far
block 11.2001.1834.1833.537
block 22.8001.321-0.909-0.688
block 33.1001.9954.3262.169
block 43.1002.4924.1011.646

reference, full softmax over all 8 scores at once: Σ softmax(s)·v = 1.646

  1. One query, eight keys. The ordinary way: find the max score, exponentiate, sum, divide, then weight the values. That needs all eight scores in hand at once. FlashAttention instead streams keys in blocks (here, two at a time) and keeps three running numbers.
  2. Block 1: scores 1.2 and −0.5. The running max becomes 1.2. Exponentiate relative to it, add to the running sum ℓ, and add the value-weighted terms to the accumulator.
  3. Block 2 holds 2.8, a new maximum. Everything accumulated so far was computed relative to the old max, so multiply ℓ and the accumulator by e1.2−2.8 to rescale them, then add the new block. Nothing earlier has to be revisited.
  4. Block 3 raises the max again, to 3.1, and the same rescale happens.
  5. Block 4: 0.7 and 2.2 are below the max, so the rescale factor is 1 and the block just adds in.
  6. Divide once at the end: acc / ℓ = 1.646, exactly the full-softmax answer. The full row of scores never existed in memory at once, which is the whole trick.
1 / 6
The online softmax at the heart of FlashAttention, with real numbers. One query's eight scores arrive two at a time. After the last block, the running result equals the full softmax-weighted sum exactly.

FlashAttention does the same FLOPs as ordinary attention (slightly more, in training) and gives the same answer up to floating-point rounding. It is faster only because it moves far fewer bytes through HBM, and it needs memory linear in the sequence length instead of quadratic. It is a perfect summary of this level: on modern hardware, the algorithm that wins is the one that respects the memory hierarchy.

So when GPT-2 prints ·floor, here is what happened under it: five token IDs looked up rows in a table; twelve layers of matrix multiplies ran as tiles staged through shared memory into tensor cores; attention streamed keys past queries in SRAM; a vector of 50,257 logits came out; a few lines of sampling code picked one. And for most of that time, the fastest arithmetic units ever built were waiting for bytes to arrive from memory.

At the bottom, LLM inference is a data-movement problem. Tensor cores make arithmetic nearly free; the engineering is in keeping them fed from a small, fast SRAM and a large, slow HBM.

Check yourselfWhat does FlashAttention reduce?

End of LLM inference · 9 levels, Idea to Silicon

Where to next

Practical notes

  1. Note3 min read

    Measure LLM tokens per second without mixing workloads

    Report prefill throughput, decode throughput and time to first token separately. A bandwidth estimate gives an optimistic bound for small-batch dense decoding.

  2. Note3 min read

    Estimate KV-cache memory before loading a model

    Compute GQA cache bytes from layer count, KV heads, head dimension and retained tokens. Use a separate latent-cache calculation for MLA models.

Browse all 34 divesAll topics