AI modelsDive 03 of 06

Mixture of experts, from router to all-to-all

Follow a token into a sparse expert layer: router scores, balance and capacity, expert parallelism, token permutation, and the memory traffic behind conditional computation.

Levels
8
Reading
42 min
Depth
Idea to Silicon
Published

A sparse layer, seen from above

Every weight is stored.
Only selected paths do work.

64 possible token–expert pairs16 selected pairsOpen the router lab

Computed illustration; these are toy scores, not measured model routes.

Level 01/08Idea

More weights, a smaller path

You have a language model that answers well, and a budget for the next token. Making every feedforward layer wider gives it more learned weights, but you must multiply by those weights again for every token you generate. Now imagine keeping several alternative feedforward networks at each layer and choosing just two for each token. You can store more learned transformations without evaluating all of them on every pass. That is the bargain behind a sparse mixture of experts.

Take the sentence The cat sat. We will follow four toy tokens through one such layer. The token cat arrives as a vector, the router picks experts, those experts transform the vector, and their weighted outputs return to the transformer. Later, we will scatter those same route records across GPUs, reorder them into matrices, and count both the work and the traffic. The numerical vectors are invented so you can check every operation. The model architectures and paper findings are real.

An A learned feedforward network chosen conditionally for an incoming token. Its weights are separate from the other experts at the same layer. is an ordinary neural network with its own weights. The name does not promise a subject specialty. The A small learned scoring function that selects which experts process a token and determines their contribution. decides which experts receive the current hidden state. Together they form one conditional layer. Different layers have different experts and routers; expert 3 at layer 5 is unrelated to expert 3 at layer 6.

Dense layer

x ∈ ℝd

One feedforward network

All its weights participate

One output vector

y ∈ ℝd

Sparse layer, top-2

x ∈ ℝd

Router selects two networks

E0 · onE1 · offE2 · onE3 · off

Weighted sum of selected outputs

y ∈ ℝd

The same feedforward interface can hide one dense network or a selection among expert networks. Both return one vector of the original model width.

Which resource are you saving?

Parameters measure how much learned state the model stores. FLOPs measure arithmetic. Bytes moved measure memory and network traffic. Latency measures the elapsed time your user experiences. Dense models couple the first two closely: double the feedforward parameters and roughly double its matrix multiplication work. Sparse layers loosen that coupling. They add their own routing, scheduling and communication costs, so the other resources still need an accounting.

Suppose one expert contains P weights, and a layer holds N experts but selects k. Its expert weights occupy NP parameter slots; one token evaluates kP of them. Increase N while holding k and expert width fixed, and the expert matmul count per token stays fixed. The router does grow with N, and the wider collection needs more memory. More capacity also needs useful training data and effective optimization. Merely adding idle experts cannot guarantee better answers.

The 2017 sparsely gated MoE paper by Shazeer et al. established this conditional-computation approach at scale, including learned sparse gates and explicit attention to load imbalance. GShard brought sparse expert layers into large transformer training with automatic sharding. These systems made the hardware part of the problem visible: a cheap arithmetic path is only useful if tokens can reach their chosen weights efficiently.

This dive begins at the feedforward boundary. If you want the surrounding transformer block, attention, or the distinction between prefill and decode, use LLM inference. Here we keep asking one narrower question: what happens when a token has several possible feedforward networks, and you evaluate only a few?

Sparsity separates stored capacity from work per token. You pay for the full collection of weights in memory and for the selected paths in computation.

Level 02/08Interface

Reading the model label

A label such as 8x7B is easy to read as eight complete seven-billion-parameter models. Mixtral shares its attention and other non-expert machinery. Only its feedforward blocks branch into eight alternatives. Its published count is 46.7 billion parameters in total and 12.9 billion active per token; the paper rounds those to 47B and 13B. You cannot get either count by multiplying the name’s two numbers.

Total means the model’s complete parameter collection. Active per token means the parameters on a token’s selected computation path, using the source’s counting convention. Attention and other always-used parts are included in that path. The active count is a useful approximation to matrix multiplication cost, but embedding lookups, attention over context, and implementation details prevent it from being a literal instruction count.

ModelTotalActive/tokenExpert selection
Mixtral 8x7B46.7B12.9B2 of 8
DeepSeekMoE 16B16.4B2.8B6 of 64 + 2 shared
DeepSeek-V3671B37B8 of 256 + 1 shared
Published whole-model counts, with selection per sparse layer. Each row links to the model’s primary source; billions use decimal units.

Count one expert before counting the model

A conventional two-matrix expert maps width d to hidden width h and back, storing approximately 2dh weights. A gated expert such as a SwiGLU feedforward network uses three matrices: an up projection, a gate projection, and a down projection. Ignoring biases, that gives 3dh. The gate projection inside this feedforward network is separate from the MoE router; the overloaded word gate describes two different operations.

Let s be the number of shared experts, each the same size as a routed expert, and let L count the sparse layers. Shared experts run for every token. The router’s dN weights are always evaluated. For three-matrix experts, the sparse part of the model has these counts:

Ptotal,sparse = L[3dh(N + s) + dN]
Pactive,sparse = L[3dh(k + s) + dN]
Fmatmul,sparse ≈ 2Pactive,sparse

Paper’s whole-model count: 46.7B total / 12.9B active.

Total sparse weights

Active sparse weights

Sparse params, total
Sparse params, active
Matmul FLOPs / token
BF16 sparse weights

Counts three bias-free matrices per expert plus routers. Excludes attention, embeddings, output head, norms, dense layers and MTP. Shared experts use the same width; FLOPs count multiply and add separately.

Change expert count or k independently. The presets reproduce the sparse-layer dimensions, while the paper’s full-model count stays separate.

Mixtral’s preset uses 32 sparse layers, d = 4,096, h = 14,336, eight experts and top-2. The DeepSeekMoE 16B preset uses its released configuration: 27 sparse layers after one dense layer, d = 2,048, h = 1,408, 64 routed experts, six selected and two shared. V3 uses 58 sparse layers after three dense layers, with d = 7,168 and h = 2,048. Those sparse counts deliberately leave the surrounding model visible as a separate cost.

What the label means when you deploy

At two bytes per stored weight, 46.7B weights alone need about 93.4 GB, or 87.0 GiB. That is before the KV cache, activations, routing buffers and runtime workspace. Active parameters do not let you load a 12.9B-sized slice once and discard everything else: the next token, or the next layer, can choose a different slice. Resident serving keeps every expert somewhere in the accelerator group.

Fine-tuning adds another distinction. Freezing base weights and training adapters reduces trainable state, but the base experts still supply the forward computation. Full fine-tuning needs gradients and optimizer state for the model being trained, distributed according to the chosen sharding plan. A single token touches a subset; a whole batch may touch every expert. The training dive covers optimizer sharding and parallelism, while quantization explains how storage precision changes the byte bill.

Check yourselfYou double the routed expert count while keeping k and each expert’s width fixed. What grows most directly?

Level 03/08Runtime

A token chooses its experts

Return to The cat sat. At the start of a sparse layer, each token already has a contextual hidden state xt of width d. Attention has helped construct that state. The router sees this vector, so its decision can depend on context even when the visible token text stays the same. It makes a fresh decision at each sparse layer.

Start with the common softmax router. A learned matrix Wr with shape N × d produces N logits. A logit is an unnormalized real-valued score. Softmax converts those scores to positive values summing to one. The router keeps the k largest scores, then normalizes their probabilities over the selected set St. This is the normalized top-k convention used in our lab.

zt = Wrxt
pt,i = exp(zt,i) / Σj=1…N exp(zt,j)
St = TopK(pt, k)
gt,i = pt,i / Σj ∈ St pt,j, for i ∈ St
yt = Σi ∈ St gt,iEi(xt)

Every expert returns a width-d vector. The weighted sum therefore fits exactly where the dense feedforward output used to go. The transformer’s residual connection surrounds this sublayer as usual. We show y for the expert branch; the residual addition remains outside the lab. No expert receives the other selected expert’s output. They process the same input independently.

With four experts and x = [1, 0], our router produces [2, 0, 1, −1]. Its softmax is approximately [0.644, 0.087, 0.237, 0.032]. Top-2 selects E0and E2, then their gate weights become approximately 0.731 and 0.269. The smaller probabilities still mattered when computing the full softmax, but their experts perform no matmul for this token.

x = [1, 0]

Rows of W and the live routing result
ExpertW rowLogitpGate
E0[2, 0]
E1[0, 2]
E2[1, 1]
E3[-1, -1]
Combined output y

Toy two-matrix ReLU experts; ties choose the lower expert index. The gate weights are normalized over the selected set.

Choose a token and change k. The lab multiplies its vector by W, computes stable softmax, selects experts, runs their toy networks, and combines the outputs.

The expert is ordinary arithmetic

The lab uses a two-matrix ReLU network small enough to inspect. For expert i, its first matrix has rows [1 + i/4, 0.5] and [−0.5, 1 + i/2]. ReLU replaces negative hidden values with zero. The second matrix has rows [1, 0.25] and [−0.25, 1]. For x = [1, 0], E0(x) returns [1, −0.25] and E2(x) returns [1.5, −0.375]. Their normalized mixture is approximately [1.134, −0.284].

Real experts use much larger matrices and often a gated nonlinearity. That changes the weight count and compute, not the routing contract. The router first decides which functions to call; the functions do useful learned transformations; a weighted reduction restores one vector per token. Thinking of experts as interchangeable functions with distinct state makes the runtime easier to follow.

A selection rule is part of the architecture

There is no universal gate formula shared by every MoE. In Switch Transformer, top-1 retains its softmax probability as the expert output multiplier. Renormalizing a single selected probability to one would remove that multiplier and its learning signal. Mixtral describes top-2 followed by softmax over the retained logits. This yields the same weights as selecting from full softmax and renormalizing, because the common denominator cancels.

DeepSeek-V3 uses sigmoid affinity scores rather than a softmax across all experts, normalizes selected affinities, and uses an additional routed-output scaling factor. We will distinguish its selection bias from these output weights at the next level. When you implement or port a model, the released gate convention matters as much as matrix dimensions.

Top-k itself is a discrete decision. Small score changes can replace one selected expert abruptly. During training, ordinary gradients flow through selected computations and gate values, while the selected index set is treated as fixed for that backward pass. A selected expert learns from its assigned tokens; an unselected expert receives no task gradient from that token through this branch. This asymmetry is where balancing becomes necessary.

The sparse layer is a learned gather, a set of independent feedforward calls, and a weighted scatter-add. Its output shape stays dense even though its computation path is sparse.

Level 04/08Runtime

Keeping the experts busy

Imagine the router sends nearly every token in our sentence to E0. E0gets frequent training updates, improves on those inputs, and may become even more attractive to the router. E3 receives little data and has fewer opportunities to improve. This feedback can concentrate traffic until most of the stored capacity is scarcely trained. At runtime, the same concentration leaves one expert’s GPU busy while the others wait.

Useful specialization requires differences between experts, but a system also needs enough work for each expert to train and enough balance to run efficiently. Perfectly uniform routing for every tiny batch can fight meaningful token preferences. Balancing methods choose how strongly, and over what population, to push traffic toward even use.

Capacity puts a hard limit on a soft preference

For T tokens, N experts and top-k routing, the mean requested load is Tk/N route records per expert. A simple capacity rule allocates C = ceil(cTk/N) slots to each expert, where c is the A multiplier on the average expert load used to size each expert’s token buffer.. For our four tokens, four experts and top-2, the average is two routes per expert. A capacity factor of one gives two slots, even if an expert receives three requests.

A capacity-limited implementation must handle the excess. Some training designs drop those expert contributions, and the residual path carries the token onward. Dropping does not delete the token from the sequence or erase its attention state. With top-2, a token can lose one route and retain another. Count dropped routes carefully before calling them dropped tokens. Switch’s top-1 makes those counts coincide for the expert branch.

A differentiable pressure toward balance

Switch defines fi as the fraction of tokens choosing expert i and Pi as the average full-softmax probability assigned to it. Its balancing term multiplies their dot product by the expert count and an external weight α. The discrete fractions are measured before capacity dropping. Gradients flow through P, treating f as fixed.

fi = (1/T) Σt 1[argmax pt = i]
Pi = (1/T) Σt pt,i
Lbalance = αN Σi=1…N fiPi

If both vectors are uniform, the unweighted term is one regardless of N. If one expert receives most hard choices, reducing its soft probability is rewarded by this objective. Do not interpret one as a universal lower bound or use the scalar alone to certify balance: the hard and soft distributions differ, and their coupling matters. Inspect the actual loads and overflow alongside the loss. The definition and scaling are from Fedus et al.’s Switch paper, equations 4–6.

Bars: requested tokens / 256. Tick: capacity. Right column: router or selection bias for the displayed batch.

Training updates
Dropped routes
Capacity / expert

A top-1 control experiment with fixed token features; only biases learn. The aux mode differentiates the Switch balancing objective. Bias mode uses sigmoid scores and a sign update; it omits V3’s node constraint and small sequence loss.

Train a biased top-1 router on 256 deterministic token feature vectors per batch. Compare the uncorrected loads, a true auxiliary-loss gradient, and selection-only bias updates.

Our experiment gives E0 a persistent initial advantage. In auxiliary mode, the learned biases are added to the logits and updated by differentiating the displayed objective. In bias mode, a separate correction affects selection scores. The experiment isolates routing: the experts and token features do not learn a language task. Switch the mode to reset the state, then train 100 batches and compare both the load shape and overflow at the same capacity.

Stability is a different problem from balance

Even a balanced router can develop awkward logit magnitudes. Softmax is unchanged if you add the same constant to all logits, so the task can tolerate offsets that make numerical behavior less comfortable. ST-MoE’s router z-loss penalizes the square of the log partition function. It gives training a reason to keep that otherwise unconstrained quantity near zero.

Logits [2, 0, 1, −1]log Σ exp(z) ≈ 2.440Squared: 5.955
Logits [12, 10, 11, 9]log Σ exp(z) ≈ 12.440Squared: 154.758

Lz = (1/T) Σt [log Σi exp(zt,i)]2

A common offset leaves probabilities and top-k selection unchanged, but changes the router z-loss. These two rows have the same relative scores.

Z-loss and load balancing act on different failure modes. One controls a logit statistic associated with stability; the other influences who gets work. Subtracting the maximum inside a numerical softmax prevents exponential overflow during that operation, but does not replace the training regularizer on the original logits. ST-MoE adds its z-loss alongside the task and balancing terms.

Let experts choose, or correct the choices

Expert Choice routing reverses the usual selection direction. Each expert picks the highest-scoring tokens up to its fixed bucket size. The expert loads are controlled by construction, while the number of experts assigned to a token can vary. Some tokens may receive several expert contributions and others none, depending on the formulation and constraints. Token-choice fixes work per token; expert-choice fixes work per expert.

SelectionFixed budgetQuantity that varies
Token-choice top-kk experts per token, before droppingTokens per expert
Expert-choiceA token bucket per expertExperts per token
Selection biask experts per tokenBiases adapt to observed load
Two ways to select entries from a token–expert score matrix. Equal total route count does not imply equal work for each token or each expert.

DeepSeek-V3 keeps token-choice but adds a bias bi to each expert’s sigmoid affinity for top-k selection. After a training batch, it decreases the bias of an overloaded expert and increases that of an underloaded one by an update speed γ. The weights multiplying the expert outputs still come from the original affinities, without this bias. Separating selection from output amplitude lets the correction shift traffic without directly rescaling an expert’s contribution.

The V3 report calls this auxiliary-loss-free balancing, and also uses a small complementary sequence-level auxiliary loss. The name describes its main balancing mechanism, not an assertion that every auxiliary term vanished. Its documented architecture uses one shared expert and 256 routed experts, selecting eight per token. It also limits a token’s routed destinations to at most four nodes. Both the load correction and the placement constraint will matter when we reach the network.

Smaller experts and a shared path

DeepSeekMoE separates two architectural ideas. Fine-grained segmentation replaces a few wide experts with more narrow ones and activates more of them under a similar expert compute budget. Shared expert isolation reserves an always-on path for transformations useful across many inputs. The routed collection can then spend less of its capacity duplicating that common work.

For a width budget example, replace eight experts of width h, selecting two, with 32 experts of width h/4, selecting eight. Both store 8h units of intermediate width and activate 2h. Their combinations, router size, dispatch records and per-expert batch shapes differ. Fine granularity gives more possible subsets; it also creates more scheduling and communication work if every selected expert is treated independently.

Input x

Shared expert
Always evaluated
Selected routed experts
Conditional weighted sum

y = Σj=1…s Eshared,j(x) + Σi ∈ S giEi(x)

Shared experts add an unconditional contribution beside the selected routed contributions. The shared path costs compute for every token.
Check yourselfAn expert overflows at capacity factor 1.0. Which action guarantees that more of its requested routes fit immediately?

Level 05/08Runtime

What an expert learns

It is tempting to label our four toy experts grammar, animals, mathematics and code. We have given them no such knowledge. Even for a trained model, a tidy subject label can tell the wrong story. Experts are optimized transformations of contextual hidden states, and a router’s preferences can align with token identity, syntax, position or many features that have no simple human name.

The Mixtral routing analysis examined assignments across The Pile subsets and layers 0, 15 and 31. Topic groups did not show a clear separation. Repeated strings and syntax, including code indentation, showed more visible routing structure, and neighboring tokens often reused experts. This supports a cautious token-pattern interpretation for the analyzed model and data; it does not establish a universal law about all MoEs.

Observation

Several topic subsets have similar expert-use distributions.

A simple one-expert-per-subject story fits poorly.

Observation

Repeated strings, indentation and adjacent tokens show routing patterns.

Local token and syntactic structure help explain the routes.

Open question

Which features does each expert actually compute?

Selection statistics alone cannot fully answer this.

Read Mixtral’s routing evidence at the scale it supports. A routing histogram measures selection, not a complete inventory of what an expert represents.

The unit of specialization is smaller than a conversation

Our cat token may choose one pair at layer 3 and another at layer 4. Another occurrence of cat, after a different prefix, may have a different hidden state and choose differently. A request does not normally select two complete experts once and stay with them throughout generation. Each layer applies its own router to the token’s evolving representation.

This explains why per-request routing summaries lose information. A document about biology can contain the same indentation, punctuation, connective words and answer templates as a mathematics document. A topic-level histogram mixes all those local choices. Conversely, similar histograms need not imply identical expert outputs: the experts still receive different hidden vectors. Counting assignments is useful, but it compresses a rich computation down to a discrete label.

How to investigate without inventing a story

Record the model checkpoint, layer, tokenization and routing convention. Compare actual token spans, not just document titles. Keep first and second choices distinct before merging them. Inspect gate weights as well as winning indices. A barely selected second expert and a dominant first expert count equally in a route histogram, despite making different weighted contributions.

Then check workloads beyond the sample that suggested the pattern. If expert E2 often receives indentation, compare ordinary text, code with different indentation, and positions near layer boundaries. If an expert looks topic-specific, test whether formatting or repeated vocabulary explains the observation. Causal interventions, such as removing or replacing an expert contribution, can add evidence about function, though they also perturb the model’s normal operating distribution.

Balance objectives complicate interpretation further. Two experts may receive similar numbers of tokens because a balancing mechanism encourages it, while learning very different transformations. An underused expert might be poorly trained, narrowly useful, or simply disadvantaged by a router that found a convenient alternative. Load alone does not identify which explanation is correct.

For serving, one finding matters even without a semantic interpretation: adjacent tokens may make correlated choices. Uniform independent routing is a useful baseline for traffic estimates, but bursts of correlated traffic can create hotspots. The runtime must use actual load measurements to place or replicate experts. A plausible specialty label cannot substitute for those measurements.

Use routing observations to explain traffic. Treat human-readable expert specialties as hypotheses that require further evidence.

Level 06/08System

The layer becomes a network

Our sentence has become four hidden vectors and eight route records. If every expert is on one GPU, routing is a local indexing problem. If the expert collection does not fit, or you want to spread its work over more devices, place different experts on different GPUs. That is Partitioning a layer’s expert collection across devices, so token activations travel to the device holding each selected expert.. The token now has a network destination as well as an expert index.

The source GPU owns the token’s place in the sequence. It sends a copy of the activation to each selected expert’s device. The expert computes a result, and the result returns to the source for the weighted sum. In the simple organization shown here, these two exchanges are called dispatch and combine. A general batch can have every device sending to every other device, producing an all-to-all pattern.

All-to-all does not mean every token visits every GPU. It means the collective exchange permits each participant to send a distinct portion to every participant. Top-k still limits each token’s expert destinations. Different tokens create the dense-looking communication matrix. The diagonal consists of local routes, which need memory operations but no inter-device transfer.

16 tokens, four per GPU. By default, GPU i hosts expert i. Each activation is 4,096 BF16 values (8,192 bytes).

Dispatch: rows send, columns receive
From / toG0G1G2G3
G0
G1
G2
G3

Remote routes
Dispatch + combine

Source tokens + expert IDs

Expert inputs = all-to-all(dispatch)

yt = Σi ∈ S(t) gt,i Ei(xt)

  1. Each source GPU packs a copy of the token vector for each selected expert.
  2. The dispatch exchange sends those vectors to the GPUs hosting their experts. Diagonal entries stay local.
  3. The combine reverses each route, transposing the traffic matrix. The source weights and sums returned vectors.
1 / 3

Payload bytes count each remote vector once per direction. Metadata, link protocol, backward pass and replication within a node are omitted.

The dispatch matrix counts routed vectors between four GPUs. Follow packing, exchange and combine; then trade expert sharding for replication and compare payload bytes.

Count vectors, then count bytes

Let T be the total tokens in an expert-parallel group, d the activation width and b bytes per element. With k routes per token, a naive dispatch carries Tkd b bytes if every route is remote. Combine carries the same amount because each expert returns a width-d vector. Let r be the fraction of route records whose destination is remote. The two directions together carry 2rTkd b bytes.

Vdispatch+combine = 2rTkdb bytes
T = 16, k = 2, d = 4,096, b = 2, r = 3/4
V = 393,216 bytes = 384 KiB

The lab’s default routes give exactly that fraction. It counts each transmitted vector once, rather than adding send-side and receive-side counters and accidentally doubling the network payload. Real protocols add route metadata, alignment and message overhead. Training adds backward communication. A link’s own byte counter can include different overheads, so first decide which quantity you want to compare.

When several selected experts live on the same remote node, an implementation can send the token to that node once and fan it out locally. Likewise, partial expert sums can sometimes be combined before crossing the slow link back. Our formula counts independent routes, making the assumptions easy to audit. Node-aware collectives can beat that payload count for the expensive tier without changing the model’s selected experts.

Parallelism choices interact

Data parallelism processes different tokens on replicas of the same weights. Tensor parallelism splits operations within a layer. Pipeline parallelism places different layers on different stages. Expert parallelism splits alternatives within a sparse layer. These dimensions can coexist, but their communication groups and memory footprints differ. The training dive covers those general mechanisms; the extra MoE question is which group holds the experts available to each token.

A dispatch exchange moves activations to different functions. An all-reduce adds corresponding partial values and distributes the result. Those are different data transformations, even if both use the same links. Training may require all-to-all for token movement and all-reduce or related collectives for replicated parameter gradients. Replacing one name with the other hides where the actual dependency lies.

GShard’s sharded transformer work made this combination concrete. GLaM also uses sparse layers within a larger language model, selecting two experts per token in its MoE layers. The architectural gain comes from conditional feedforward work; turning it into throughput depends on how those layers are partitioned and fed.

Limit destinations at the slow boundary

DeepSeek-V3 groups experts across nodes and limits each token to at most four destination nodes during training. It still selects eight routed experts, so a node may handle several choices. The limit controls traffic fan-out across the slower network tier while preserving selection among many experts. Its reported implementation places the routed experts of each layer across 64 GPUs in eight nodes. That is a specific training topology, not a requirement for all V3 deployments.

For inference, batching gives the runtime more tokens to pack into each expert’s input matrix. A larger batch improves weight reuse, but it also creates more route records and can add queueing delay. The right batch policy balances throughput and response latency. An expert receiving one token cannot borrow the arithmetic reuse of a different expert receiving a hundred, because they multiply by different weight matrices.

Offloading stores some experts in host memory or another slower tier and transfers selected weights when needed. It can make an otherwise oversized model runnable. A token then waits for weights or relies on accurate prefetch and a useful resident cache. Correlated routes may help reuse, while a diverse batch may touch nearly the whole collection. Active-parameter counts do not capture these transfer stalls.

Check yourselfIn the lab’s default top-2 configuration, 24 routes are remote. Each vector is 8 KiB. What is dispatch plus combine payload?

Level 07/08Machine

Turn a routing list into matmuls

The network delivers token vectors, but a GPU kernel wants regular matrices. Our router’s output is a list of records such as token 2, expert 0, gate 0.5. In sequence order, successive records can belong to different experts. Launching a separate tiny matrix multiply for every record throws away batching and repeats setup work. First rearrange the records so each expert receives a contiguous set of rows.

A useful route record preserves three things: the original token index, the destination expert index and the combine weight. For top-k, each token contributes k records. Sort or bucket those records by expert, gather the token vectors in that order, and record the segment boundaries. You now have matrices X0, X1, and so on, with potentially different row counts mi.

Token order: route records

t0 · TheE0g = 0.731
t0 · TheE2g = 0.269
t1 · catE1g = 0.731
t1 · catE2g = 0.269
t2 · satE0g = 0.500
t2 · satE1g = 0.500
t3 · .E1g = 0.622
t3 · .E3g = 0.378

Expert order: contiguous groups

t0 · TheE0g = 0.731
t2 · satE0g = 0.500
t1 · catE1g = 0.731
t2 · satE1g = 0.500
t3 · .E1g = 0.622
t0 · TheE2g = 0.269
t1 · catE2g = 0.269
t3 · .E3g = 0.378

Expert order: computed outputs

t0 · TheE0[1.00, -0.25]
t2 · satE0[1.63, 0.13]
t1 · catE1[0.88, 1.38]
t2 · satE1[2.00, 0.56]
t3 · .E1[0.31, 1.25]
t0 · TheE2[1.50, -0.38]
t1 · catE2[1.00, 1.88]
t3 · .E3[0.44, 1.75]

Token order: weighted scatter-add

t0 · They[1.134, -0.284]
t1 · caty[0.909, 1.509]
t2 · saty[1.813, 0.344]
t3 · .y[0.360, 1.439]
  1. Top-2 creates eight route records for four tokens. Keep the token index and gate beside each route.
  2. Sort route records by expert. Each expert can now read one contiguous input matrix.
  3. Run the different expert matmuls as a group. These outputs come from the same toy ReLU experts as the router lab.
  4. Use the saved token index to scatter each output back, multiply by its gate, and add the two contributions.
1 / 4

Same routes, three allocation rules

E0: 2 rows

xx·

E1: 3 rows

xxx

E2: 2 rows

xx·

E3: 1 rows

x··
Useful rows
8
Uniform padding
12
Two-row tiles
10

Dots are uniform padding. Two-row tiles illustrate block rounding; actual MegaBlocks tiles and kernels are larger. Dropless execution retains all eight routes.

The four-token top-2 batch, computed with the router lab’s exact toy experts. Permutation changes row order; scatter-add restores the token order and combines duplicate token records.

Grouped GEMM shares scheduling, not weights

GEMM is general matrix multiplication. A grouped GEMM launches or schedules a collection of matmuls together, often with different row counts. Each expert still owns its own weights. Grouping gives the device a larger set of tiles to schedule and reduces overhead compared with treating every tiny expert call independently. It does not turn the experts into a single shared weight matrix.

For each expert, a gated feedforward pass has two d-to-h projections, an elementwise gated nonlinearity, and one h-to-d projection. The runtime may fuse some of those operations or store intermediates differently. The logical sequence remains a set of per-expert computations followed by a reduction by original token index. The saved gate travels with the record throughout.

For our The token, the two sorted records are separated into E0’s and E2’s groups. After both groups finish, scatter-add multiplies each result by its own gate and accumulates both into y0. A plain inverse permutation of eight rows would still leave eight rows. The final reduction is what brings the top-2 representation back to four token outputs.

Padding trades simplicity for empty work

A fixed-capacity layout can give every expert a matrix with C rows. The data structure is convenient, but an expert with mi < C has empty slots. Kernels that compute those padded rows spend arithmetic on zeros or masked values. If mi > C, the implementation must overflow, reroute, allocate more space, or drop contributions. Capacity, useful arithmetic and executed arithmetic are distinct quantities.

For three-matrix experts, each useful route costs approximately 6dh floating-point operations, counting a multiply and add as two. Summing mi over the experts gives Tk before dropping. With no dropping, useful expert matmul work is 6Tkdh. Uniform padded execution instead performs approximately 6NCdh if every capacity row reaches the matmuls. A good load histogram helps, but even balanced small matrices need hardware-friendly tile shapes.

MegaBlocks reformulates expert computation using block-sparse operations to avoid the usual choice between dropping routes and padding every expert to a common capacity. It packs the routed computation into regular blocks that efficient GPU kernels can process. Dropless means all assigned token contributions are retained. It does not mean arbitrary matrix dimensions magically fit the hardware without any block rounding or bookkeeping.

The small allocation figure uses two-row blocks to make rounding visible. It compares useful rows, padding to the largest expert load, and rounding each nonempty expert group to a block multiple. Actual MegaBlocks block dimensions are larger. The principle is that empty capacity for one expert should not force equal empty work across every expert.

The logical combine after grouped expert computation
for (const record of routesSortedByExpert) {
  const value = expertOutput[record.sortedRow];
  for (let j = 0; j < modelWidth; j++) {
    output[record.token][j] += record.gate * value[j];
  }
}

A production GPU combine parallelizes this loop and resolves concurrent contributions with an appropriate reduction strategy. The order of floating-point additions can change low bits. What must survive every optimization is the mapping: each expert output returns to its original token with the correct gate. Sorting by expert is a storage transformation, not permission to reorder the sequence seen by attention.

The router itself also costs a dense matrix multiplication: T tokens of width d times N router columns, roughly 2TdN FLOPs. Add top-k selection, permutation, gather and scatter traffic, nonlinearity, and communication to the expert matmul count. Same active parameters therefore does not promise identical elapsed time between a dense model and an MoE. At small sizes these overheads can dominate; at large sizes memory and links may dominate instead.

Efficient MoE kernels make a dynamic routing list look regular enough for matrix hardware. Keep useful rows, padded rows and dropped routes separate in every performance claim.

Level 08/08Silicon

The price of a small batch

We have put every token beside the right expert and turned its inputs into matrices. Why can the GPU still look underused? Matrix hardware becomes efficient when it reuses weights across many rows. Sparse routing divides the token batch among experts, and small decode batches can leave each expert with only one or two rows. The GPU then spends much of its time fetching weights for very little arithmetic.

Consider one projection Y = XW with X shaped m × d and W shaped d × h. It performs about 2mdh FLOPs. In an ideal implementation, W is read once, X is read once and Y is written once. With b bytes for every element, the lower-bound traffic is b(dh + md + mh). Their ratio is Floating-point operations divided by bytes transferred at the memory boundary being analyzed.. Specify the memory boundary: here we are estimating traffic to high-bandwidth device memory.

I(m) ≈ 2mdh / [b(dh + md + mh)] FLOPs/byte
For small m: I(m) ≈ 2m/b
Balanced mean expert rows: m ≈ Bk/N

For BF16, b = 2, so the small-m approximation is m FLOPs per byte. With B = 32, N = 8 and k = 2, each expert sees eight rows on average. A dense projection with the same matrix dimensions sees all 32 rows. Those are very different opportunities to reuse a fetched weight, even though the MoE has selected only a subset of its total collection for each token.

Dense: B rows
MoE: Bk/N rows per expert
Mean rows / expert
MoE ridge batch
Expected touched experts

One 4,096 × 14,336 projection; 8 experts, top-2. Ideal reuse, balanced mean load and one weight read per matmul. Expected touched experts assumes independent uniform top-2 sets across tokens. Bars saturate at the selected ridge.

Compare a dense projection with the mean expert projection at the same decode batch. Increase B until the average expert has enough reuse to reach the chosen roofline ridge.

A roofline threshold is a ratio

The ridge point is the ratio of sustained compute throughput to sustained memory bandwidth, expressed in FLOPs per byte. Below it, an idealized kernel is limited by memory traffic. Above it, compute can become the limiting resource. The inference dive develops the roofline model and GPU hierarchy. Here the new effect is the division of the batch by N/k.

The slider chooses a hypothetical ridge rather than borrowing a peak number from a particular accelerator. Peak tensor throughput and achieved HBM bandwidth are not interchangeable with a measured application roofline. The calculator solves the full displayed formula for the row count at that ridge and converts it back to a batch using B = mN/k. Real routing produces integer loads around that mean, including empty experts for small B.

Under independent uniform top-k choices, one expert is missed by a token with probability 1 − k/N. It is missed by all B tokens with probability (1 − k/N)B. The expected number of distinct experts touched by the batch is N[1 − (1 − k/N)B]. For eight experts and top-2, even a modest batch tends to touch nearly the whole collection. A single token’s sparse path should not be confused with a whole batch reading only that fraction of weights.

This also explains an apparent reversal. Small batches benefit from skipping most expert weights but suffer low reuse in the experts they do touch. Large batches can keep expert matmuls busy, yet collectively access almost every expert. Throughput improves by amortizing the full weight reads over more tokens. Total weight memory remains large in either regime.

The other bandwidth boundary is between GPUs

Device memory supplies the expert weights; the interconnect supplies remote token vectors. These are different streams. Increasing batch size can improve expert weight reuse while making the dispatch exchange larger. The next bottleneck may move from HBM traffic to network bandwidth, synchronization, or a hotspot expert. A useful profile separates those phases instead of reporting just average GPU utilization.

Inside one accelerator

HBM ↔ matrix hardware

Expert weights and local activations

Reuse rises with mi

Across accelerators

Source GPU ↔ expert GPU

Dispatch and return vectors

Payload ≈ 2rTkdb

The two bandwidth budgets carry different objects. Weight traffic grows with the expert matrices touched; network traffic grows with remotely routed activation vectors.

Large-memory nodes keep more experts resident. High HBM bandwidth helps stream large expert matrices. Fast links within a node let dispatch reach several devices cheaply, and node-limited routing controls how often the slower tier participates. Those traits make some multi-accelerator nodes a comfortable home for MoE, but topology still needs to match the actual batch and placement plan.

Lower precision reduces stored weight bytes and can raise arithmetic intensity, though real quantized kernels introduce scales, packing and conversion work. The quantization dive follows that representation change. Our one-byte option is an idealized FP8 calculation in which weights and activations both occupy one byte. It is a way to inspect the denominator, not a prediction that a particular quantizer doubles serving throughput.

When serving stalls, trace our token one last time. Did the router send it to an overloaded expert? Did capacity drop its contribution? Did dispatch wait on a slow link? Did permutation spend longer moving rows than the matmul spent computing? Did a one-row expert stream hundreds of megabytes of weights? Did an offloaded expert arrive late? Each symptom belongs to a specific step of the sparse layer, and each calls for a different fix.

The attractive part of MoE survives all this accounting: the model can retain many learned feedforward transformations while evaluating only a few for a token. The engineering work makes that selected path economical. Its cost is a product of routing, batches, placement and kernels, rather than the active-parameter number alone.

Check yourselfWith balanced top-2 routing to 8 experts, which batch gives a mean of 32 rows per expert?

A sparse path saves arithmetic. Real speed comes from enough rows per expert, resident weights, and a communication plan that fits the machine.

Primary sources

End of Mixture of experts · 8 levels, Idea to Silicon

Where to next

Browse all 34 divesAll topics