AI servingDive 02 of 06
Quantization, from 16 bits to 4
Squeeze a 70-billion-weight model into fewer bytes. Explore the floating-point encodings, tame outliers, compensate rounding errors, then unpack four-bit weights inside a GPU kernel.
- Levels
- 8
- Reading
- 40 min
- Depth
- Idea to Silicon
- Published
70 billion weights · the same model, a smaller alphabet
Decimal GB. Payload only; a real file also stores scales, metadata and some higher-precision tensors.
A smaller alphabet for weights
You have a model with 70 billion weights and a machine with 48 GB of accelerator memory. Each weight is a BF16 number, two bytes. The weights alone occupy about 140 GB. Before you reserve space for a conversation, a temporary buffer or a single output token, you are short by almost a hundred gigabytes.
Weights are learned coefficients in the model's matrices. Activations are the intermediate vectors those matrices transform while processing your input. Weights persist across requests; activations depend on the tokens being processed.
Four bits per weight changes the arithmetic. The raw payload becomes 35 GB. Scales and other metadata bring it back up, and some tensors may stay in a wider format, but suddenly the model might fit. You have changed the representation of its numbers rather than its architecture. The same matrix shapes, layers and vocabulary remain; the weight values have moved a little.
That is Representing values with a smaller set of allowed reconstruction levels, usually with a scale that maps those levels to the original range.. A real-valued weight such as 0.1372 becomes a nearby value from a finite alphabet. BF16 already chooses from a finite alphabet. Low-bit quantization makes that alphabet much smaller, and usually gives a group of weights a shared ruler so its few available levels land somewhere useful.
Capacity is only half the attraction. At small batch sizes, decoding repeatedly reads nearly all the weights of a dense model. Fewer bytes can shorten those reads. The LLM inference dive's roofline explains why that workload is often limited by memory bandwidth. Here we will keep the serving model simple and follow the numbers being streamed: what four bits can represent, how you choose them, and how a kernel turns them back into useful operands.
What you give up
Consider one multiply: a weight of 0.1372 times an activation of 3. If the reconstructed weight is 0.14, the output changes by 0.0084. A layer sums thousands of such products. Errors may partly cancel, or they may reinforce one another. Later nonlinearities can suppress a difference or turn it into a different next-token choice. There is no universal number of bits at which every model stays good.
Why does the idea work at all? Learned models often have room to move. Many weights occupy a fairly narrow range; the exact final digits need not be crucial to the function. Training also produces correlated features, so a perturbation in one weight can sometimes be offset by a perturbation elsewhere. Sensitivity varies: some channels have large activations, and small changes can matter greatly when two output choices are close.
The useful question is therefore about outputs: how much does a compressed layer's answer change on inputs you care about? Measuring only the difference between the original and reconstructed weights misses that distinction. We will start with nearest-value rounding, then use calibration inputs to make a more informed choice.
Keep the 70-billion-weight model in mind. We'll use small synthetic tensors to trace float encoding, rounding compensation and block decoding, with calculations you can check.
Several original weights could produce that same code. Quantization discards those differences permanently. You keep a nearby value and accept its effect on the computation. For the 0.1372 example the absolute weight error is 0.0028, about 2% of the original magnitude. Its effect depends on the surrounding layer.
Quantization buys memory and sometimes speed by spending numerical accuracy. Judge the result by the model's useful behavior.
What a quantized checkpoint promises
The first obstacle is the file name. A model download might offer BF16, Q8_0, Q4_K_M, GPTQ, AWQ and NF4. These names sit at different layers. Some describe a numeric encoding, some an algorithm that chooses the numbers, and some a container or a recipe mixing tensor types. Treating them as interchangeable versions of “four-bit” hides the details that determine whether your runtime can load them.
GGUF is a container: it stores tensors and metadata. Q8_0 and Q4_K are ggml tensor encodings. A whole-file label such as Q4_K_M describes a quantization recipe that can use different encodings for different tensors. Inspect the tensor inventory to find the exact file size and effective bits per parameter.
| What you see | What it describes | What runs it | What to inspect |
|---|---|---|---|
| GGUF: Q8_0, Q4_K_M | A container plus a tensor-type recipe | llama.cpp and compatible runtimes | Which tensors use which block types? |
| GPTQ checkpoint | Weights fitted with calibration inputs | A matching GPTQ kernel and layout | Bits, group size, zero-point and column order? |
| AWQ checkpoint | Activation-aware scaling and low-bit weights | A matching AWQ packing and kernel | Which layers and quantization configuration? |
| FP8 serving | FP8 weights and often FP8 activations | Supported FP8 kernels, e.g. Hopper | E4M3 or E5M2, scales static or dynamic? |
| bitsandbytes NF4 | Block-scaled four-bit lookup codes | bitsandbytes-compatible dequantization | Compute dtype, block size, double quantization? |
Read W4A16 as two promises
W4A16 means four-bit weights and 16-bit activations. It usually describes weight-only quantization: the weights are compact in memory while the activations stay in FP16 or BF16. Check the codebook, group size, scale dtype and packing, along with the kernel's accumulator type. Two W4A16 checkpoints can be incompatible even if both contain exactly four payload bits per weight.
W8A8 quantizes weights and activations. Both inputs to a matrix multiply can then use an eight-bit arithmetic path, often with wider accumulation. INT8 and FP8 are different ways to do this. A label that says only “8-bit” leaves out whether the operation uses an integer dot product or floating-point multiplication, and how values are rescaled afterward.
A practical W4A16 kernel might unpack integer weight codes into FP16 register fragments, multiply them by FP16 activations on tensor cores, and accumulate into FP32. A native low-bit floating-point kernel may instead consume FP4 operands with block scales directly. The storage dtype, multiply dtype and accumulator dtype are three separate choices. The kernel determines which arithmetic instructions run.
Weights are fixed; activations arrive later
You can quantize a weight matrix once when preparing a checkpoint. Activations depend on the prompt, the token and the batch. Their scales can be calibrated in advance, computed dynamically, or chosen at a finer granularity inside a kernel. Static scales save runtime work but risk clipping inputs outside the calibration range. Dynamic scales adapt, at the cost of finding a range and transforming values during execution.
The KV cache is a third target. It stores activation-derived keys and values across decoding steps, so reducing its precision can relieve a growing memory budget even when the weights already fit. The inference dive derives its size. Cache precision is configured separately: a server may run W4A16 with a BF16 cache, or FP8 weights with an FP8 cache.
Cache quantization has its own sensitivity. An error in a key can change attention scores and therefore which values are mixed; an error in a value changes the mixture directly. Long contexts, head layouts and the attention kernel all matter. Evaluate cache precision as a distinct setting, with the same prompts and context lengths you intend to serve.
A useful order for choosing
Start with the runtime and hardware you have. Find the layouts its optimized kernels consume, then choose a checkpoint and quality budget within that set. Check group size, scale dtype, zero-point convention, skipped modules and any activation reordering. Finally measure loading, prefill and decode separately. A mathematically good quantizer with an unsupported layout is still a conversion job waiting to happen.
Where the floating point goes
Before choosing which values to keep, look at what a floating-point value is. Binary scientific notation writes a normal number as a sign, a leading 1 with a fractional tail, and a power of two. The stored fields are a sign bit S, an exponent E and a fraction M. If the fraction has m bits and the exponent bias is b, the value is:
normal: (−1)ˢ × (1 + M / 2ᵐ) × 2ᴱ⁻ᵇ
The leading 1 is implicit. FP16 stores ten fraction bits but normally carries eleven significant binary digits. “Mantissa bits” in format names usually counts the stored fraction. The bias lets an unsigned exponent field represent powers above and below zero: FP16's stored exponent 15 means 2⁰.
For example, 1.5 is 1.1₂ × 2⁰. In FP16 it has sign 0, stored exponent 01111 and fraction 1000000000, producing 0x3E00. The fraction contributes one half. Move the exponent up by one with the fraction unchanged and the value becomes 3. This field-by-field interpretation is what the explorer applies when you flip a bit.
At exponent zero the leading 1 disappears. These Values with the minimum exponent and no implicit leading one. They fill the gap between zero and the smallest normal value at constant absolute spacing. values use M/2ᵐ × 2¹⁻ᵇ. All-zero exponent and fraction encode zero; the sign bit distinguishes +0 and −0. In the IEEE binary formats, an all-ones exponent encodes infinity when the fraction is zero and NaN otherwise. IEEE 754-2019 defines binary16 and binary32 and their arithmetic conventions.
Think of the exponent as choosing a neighborhood and the fraction as choosing an address within it. Each power-of-two interval has the same number of addresses. The addresses spread farther apart as the neighborhood gets larger. This is why floating point gives roughly relative precision while a fixed-scale integer ruler gives constant absolute spacing.
- Sign
- Exponent
- Fraction
Nearest, ties to even. FP8 and FP4 conversions saturate; FP32, FP16 and BF16 overflow to infinity. Type Infinity, -Infinity, NaN or -0. Click bits to decode any pattern.
FP16 spends on precision; BF16 spends on range
Bfloat16's definition keeps FP32's eight-bit exponent and bias of 127, but keeps only seven fraction bits. It has the same normal exponent range as FP32, with fewer values between powers of two. FP16 instead has five exponent bits and ten fraction bits. Around 1, adjacent normal values are separated by 1/128 in BF16 and 1/1024 in FP16. BF16 reaches much larger magnitudes; FP16 distinguishes closer neighbors in this neighborhood.
The maximum finite values of BF16 and FP32 are slightly different because their largest significands differ. The explorer includes BF16 subnormals, which TPU conversion flushes to zero.
| Format | Sign / exponent / fraction | Bias | Smallest positive normal | Smallest positive subnormal | Largest finite |
|---|---|---|---|---|---|
| FP32 / binary32 | 1 / 8 / 23 | 127 | 2⁻¹²⁶ | 2⁻¹⁴⁹ | ≈3.403 × 10³⁸ |
| FP16 / binary16 | 1 / 5 / 10 | 15 | 2⁻¹⁴ | 2⁻²⁴ | 65,504 |
| BF16 | 1 / 8 / 7 | 127 | 2⁻¹²⁶ | 2⁻¹³³ | ≈3.390 × 10³⁸ |
| OCP E4M3 | 1 / 4 / 3 | 7 | 2⁻⁶ | 2⁻⁹ | 448 |
| OCP E5M2 | 1 / 5 / 2 | 15 | 2⁻¹⁴ | 2⁻¹⁶ | 57,344 |
| E2M1 | 1 / 2 / 1 | 1 | 1 | 0.5 | 6 |
FP8 makes the trade visible
The OCP 8-bit Floating Point Specification, revision 1.0 with the December 2023 bias correction, defines E4M3 and E5M2. E4M3 has a bias of 7 and maximum finite magnitude 448. It has no infinity: exponent 1111 still holds finite values except fraction 111, which denotes NaN. There is one NaN exponent/fraction pattern, with either sign, giving byte encodings 0x7F and 0xFF.
E5M2 uses bias 15, reaches 57,344, and reserves the all-ones exponent for infinities and NaNs. Its wider range costs a fraction bit. With only two fraction bits, the normal values in [1, 2) are 1, 1.25, 1.5 and 1.75; E4M3 fits eight. FP8 conversions also need an overflow policy. OCP specifies saturating and non-saturating modes; the explorer uses saturation for FP8, so a large finite input lands on the largest finite value.
At four bits, OCP's E2M1 has just eight nonnegative magnitudes: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. The sign bit supplies their negatives, including negative zero. There is no NaN or infinity. Raw E2M1 cannot represent a weight of 0.1372 usefully. Its smallest nonzero magnitude is 0.5. A shared scale is what turns this tiny alphabet into a practical tensor format.
Let a block share the exponent budget
The OCP Microscaling Formats specification v1.0 defines MXFP8 (E4M3 or E5M2), MXFP6 (E3M2 or E2M3), MXFP4 (E2M1) and MXINT8. All use 32 elements per block and an eight-bit E8M0 scale. E8M0 has no sign or fraction: code c represents 2ᶜ⁻¹²⁷ for c from 0 through 254. Code 255 is NaN; zero and infinity have no encoding. MXINT8 elements are signed fixed-point values with six fractional bits.
For MXFP4, 32 four-bit codes plus one eight-bit scale occupy 136 bits: 4.25 bits per value. The reconstructed element is its E2M1 value times the shared power of two. NVIDIA's NVFP4 instead shares an E4M3 FP8 scale across 16 E2M1 elements and adds a per-tensor FP32 scale. That costs 4.5 bits per value before the tensor-scale overhead. A fractional block scale gives more freedom than a power of two.
- Mean squared error
- Bits / value before tensor scale
Input above reconstruction. MX chooses the smallest power-of-two scale that avoids overflow; NVFP4 rounds amax/(6 × tensor scale) to E4M3. These are illustrative scale-selection recipes.
A low-bit float is useful together with its scale. Count the whole block, and distinguish the element encoding from the rule that chooses its scale.
A ruler, a scale and an outlier
An integer quantizer uses a simpler ruler. Choose a step size s and an integer zero-point z. Divide a real value by s, round to an integer, shift by z, then clamp to the allowed code range. To reconstruct, subtract z and multiply by s:
q = clamp(round(x / s) + z, qmin, qmax)
x̂ = s × (q − z)
With s = 0.02 and z = 0, our weight 0.1372 becomes q = 7 and reconstructs as 0.14. The error is 0.0028. If the range contains the input and the scale is exact, round-to-nearest bounds the absolute error by s/2. That guarantee ends at the clipping boundary. A value outside the selected range can be arbitrarily far from the endpoint it gets clamped to.
Symmetric quantization centers the ruler at zero. A common INT8 convention uses −127 through 127, leaving −128 unused so both signs have equal endpoints. The tensor lab uses the analogous symmetric range at every bit width: −7 through 7 at four bits. Other formats use all 16 codes, so check the encoding's integer range.
Asymmetric, or affine, quantization lets the zero-point move. For unsigned b-bit codes, a simple min/max recipe takes s = (max − min)/(2ᵇ − 1) and z = round(−min/s), with the range expanded to include zero. Exact zero is useful for padding and sparse values. If the distribution is mostly positive, an affine ruler avoids spending half the codes on a negative range you do not need. Rounding the zero-point can shift the endpoints slightly.
Choosing the ruler's range is already an optimization. Suppose 255 ordinary weights sit between −0.2 and 0.2 while one is 1.2. A symmetric four-bit ruler spanning ±1.2 has a step near 0.1714; most ordinary weights collapse to −0.1714, zero or 0.1714. Clipping the range to ±0.2 gives them a step near 0.0286, but reconstructs the outlier as 0.2 instead of 1.2. That one error has squared magnitude 1, which may overwhelm the improvements elsewhere.
You can search clipping thresholds to minimize an error objective rather than always using the extrema. Whether the outlier should dominate that objective depends on what multiplies it. Activation-aware methods use those inputs to judge which reconstruction mistakes are expensive for the layer.
How many values share the ruler?
A per-tensor scale is cheap: one scale covers the matrix. It is also held hostage by the largest magnitude in that matrix. Per-channel scales give rows or columns independent ranges, with the axis specified by the operator. Per-group scales divide a channel into shorter runs, often along the input dimension of a weight matrix. “Group size 128” means little until you know the axis and storage order.
Smaller groups tend to reduce range mismatch, but they add scales, sometimes zero-points, and more work in the kernel. With four payload bits and one FP16 scale per 32 weights, storage is 4 + 16/32 = 4.5 bits per weight. At group size 128 the same scale overhead is only 0.125 bits per weight. A smaller group can make a better approximation while making a larger file.
- Input histogram
- Reconstructed histogram
- Mean squared error
- Signal / noise
- Bits / weight
First group's scale and reconstruction levels
Seeded Gaussian samples (σ = 0.12), with optional ±1.2 outliers; generated here, not extracted from a model. Overhead assumes one FP16 scale and, for affine, one byte zero-point per group.
The histogram shows where reconstructed values land. In the four-bit, whole-tensor case, a few outliers stretch the scale so much that many ordinary weights share the same few codes. Remove the outliers or shorten the groups and those weights gain more distinct reconstruction levels.
The mean squared error is the average of (x − x̂)². The signal-to-noise ratio shown here is 10 log₁₀(mean(x²)/MSE), in decibels. Higher is better for this reconstruction metric. It is possible to improve this number while worsening an important model output, because the metric gives every weight equal importance and never looks at the inputs that multiply it.
Outliers cluster in channels
LLM.int8(), Dettmers and colleagues, 2022, identifies large, systematic activation features in large transformers. A few feature dimensions can dominate the range. The method uses vector-wise INT8 quantization for most of the matrix multiply and separates outlier feature dimensions into a higher-precision FP16 multiplication. The partial results are added. It preserves the difficult features instead of forcing them onto the same INT8 ruler as the rest.
SmoothQuant, Xiao and colleagues, takes a different route to W8A8. Write a channel scale vector s into a diagonal matrix D. For a row-major activation matrix, XW = (XD⁻¹)(DW): divide each activation channel by s and multiply the corresponding weight channel by s. Before quantization, the layer's function is unchanged. The transformation moves some of the activation range into the fixed weights, where quantization is easier to prepare offline.
The paper chooses scales using activation and weight channel maxima, with a parameter α controlling how much difficulty moves to the weights. At α = 0.5, sⱼ = sqrt(max|Xⱼ|/max|Wⱼ|). Shrinking an activation channel expands its corresponding weight range, balancing the two quantization errors.
Four activation channels, three calibration tokens. The first three channels stay near 1; the fourth grows with the slider.
| W8A8 path | Output squared error |
|---|
Per-channel activation scaling gives an excellent numerical reference, but a different scale along the reduction dimension is awkward for a single fast integer matrix multiply. SmoothQuant aims to keep a hardware-friendly activation quantization path while reducing that mismatch. On this tiny example you can inspect the trade directly; the best setting on a real layer needs representative calibration data.
NF4 spends more codes near zero
Four bits can also index a nonuniform alphabet. QLoRA, Dettmers and colleagues, 2023, introduces NormalFloat4, or NF4, whose levels are derived from normal-distribution quantiles, normalized to [−1, 1], with an exact zero. Each stored nibble indexes a lookup table. NF4 assigns more reconstruction levels near zero, where roughly normal weights tend to cluster.
A block scale expands that normalized table to the block's range. A weight near our running 0.1372 example can map to a useful nearby value if the block scale places the NF4 levels appropriately. Outliers still matter: they can enlarge the block's normalization range and move its dense central levels away from small weights.
QLoRA keeps the quantized base weights frozen and trains low-rank adapters through dequantized computations. It also uses double quantization, quantizing the quantization constants themselves to reduce metadata. The bitsandbytes implementation uses sixteen fixed lookup values, block state and a wider compute path.
Granularity and outlier handling often matter as much as bit width. A smaller code alphabet works best when its scale follows the values that share it.
Round for the layer, not the weight
Now give the quantizer a layer's inputs. Use the column-vector convention Y = WX: each column of X is one calibration input, and each row of W produces one output channel. Replacing W with Ŵ creates output error (W − Ŵ)X. Rather than minimizing the distance between weights, minimize the squared Frobenius norm of that output error:
L = ‖(W − Ŵ)X‖²ꜰ
H = 2XXᵀ
The Hessian H describes the curvature of this quadratic loss for one weight row. Its diagonal measures how strongly each input feature is exercised. Off-diagonal entries capture feature correlations. If two input dimensions tend to move together, increasing one weight and decreasing another can partly preserve the output. Independent rounding ignores that opportunity.
GPTQ: fix a weight, repair the rest
GPTQ, Frantar and colleagues, 2022, uses approximate second-order information to quantize weights while compensating their errors. For the elementary optimal-brain-surgeon update, round weight wᵢ to qᵢ, then change the still-free weights using the inverse Hessian of the remaining problem:
wⱼ ← wⱼ − ((wᵢ − qᵢ) / [H⁻¹]ᵢᵢ) × [H⁻¹]ᵢⱼ
The sign matters. If rounding moves one weight up, the correlated weights may need to move down. Each fixed column becomes unavailable for further repairs. After eliminating it, the inverse Hessian for the reduced problem changes too; repeatedly using the original inverse would be wrong. Production GPTQ uses a Cholesky-based formulation and block updates to do this efficiently across a layer, usually with damping for stability.
A calibration matrix can have redundant or inactive input features, making XXᵀ singular or poorly conditioned. Adding a small positive diagonal term before factorization stabilizes the update, at the cost of changing its objective. A million nearly identical prompts can still leave an important input direction untested. Choose calibration examples that exercise the kinds of text and sequence lengths the model will encounter.
Here is a complete two-weight calculation. W = [0.6, 0.6] and X has rows [1, 0] and [1, 1]. The two calibration inputs are its columns, so the original outputs are [1.2, 0.6]. We deliberately choose a fixed unit-step grid to keep the calculation visible. Both 0.6 weights independently round to 1; the outputs become [2, 1], with total squared error 0.8.
For these inputs, H = [[2, 2], [2, 4]], whose inverse is [[1, −0.5], [−0.5, 0.5]]. Fixing the first weight at 1 creates w₁ − q₁ = −0.4. The update lowers the second weight to 0.4. Round that to zero and the row becomes [1, 0]. Its weight error is larger than independent rounding's, but its output error is smaller.
W = [0.6, 0.6] · X = [[1, 0], [1, 1]] · grid = integers
Original row
[0.6, 0.6]
- Output ŴX
- [1.2, 0.6]
- Compensated output error
- 0
- Nearest-rounding output error
- 0.8
First column fixed, second compensated
[1, 0.4]
- Output ŴX
- [1.4, 0.4]
- Compensated output error
- 0.08
- Nearest-rounding output error
- 0.8
Both columns fixed
[1, 0]
- Output ŴX
- [1, 0]
- Compensated output error
- 0.4
- Nearest-rounding output error
- 0.8
- Two calibration inputs are the columns of X: [1, 1] and [0, 1]. Original outputs are [1.2, 0.6]. Round-to-nearest alone gives [1, 1], outputs [2, 1] and squared error 0.8.
- Round 0.6 to 1. With H⁻¹ = [[1, −0.5], [−0.5, 0.5]], update the second weight: 0.6 − ((0.6 − 1) / 1) × (−0.5) = 0.4.
- Round the compensated 0.4 to 0. The final row [1, 0] gives squared error 0.4, half of independent rounding's 0.8 on these inputs.
Exact two-column OBS update underlying GPTQ; fixed unit grid, no clipping or damping. Production GPTQ batches updates and uses a Cholesky factor.
Calibration determines which input directions the optimization protects. Inputs absent from that sample may exercise the layer differently, and later layers change the relationship between local error and final behavior.
AWQ: spend precision where activations say it matters
AWQ, Lin and colleagues, 2023, also looks at activations, but protects salient weight channels with scaling rather than GPTQ's sequential error compensation. A large weight is not necessarily important; a channel multiplied by large activations can be sensitive even when its weights look ordinary. AWQ searches channel scales using calibration activations and evaluates the resulting quantized layer.
Scale an important weight channel up before quantization and compensate by scaling its input down. After undoing that transform, a given quantization step corresponds to a finer step in the original weight units. Shared group scales limit the gain because enlarged channels can stretch their neighbors' range. AWQ searches for a useful balance while keeping the protected channels quantized.
k-quants compress the rulers too
Algorithms choose values; block layouts decide how to store them. In llama.cpp's Q4_K tensor type, one super-block contains 256 weights split into eight groups of 32. Each group gets a scale and a minimum coefficient, both stored in six bits and expanded by two FP16 super-scales. Reconstruct a weight as d × scale × q − dmin × min. The minimum coefficients encode the subtractive offsets.
Q4_K · one 256-weight super-block
d, dmin4 bytes- Two FP16 super-scales
scales12 bytes- 8 scales + 8 minima, each 6 bits
qs128 bytes- 256 four-bit weight codes
4 + 12 + 128 = 144 bytes · 144 × 8 / 256 = 4.5 bits/weight
The source's structure and size assertion give 4 bytes of FP16 super-scales, 12 bytes of packed scale/minimum coefficients and 128 bytes of nibbles: 144 bytes for 256 weights, or 4.5 bits per weight. The dequantization function confirms the reconstruction rule. Q4_K_M mixes tensor types, so its average depends on the tensor inventory.
Training can adapt; evaluation must catch what remains
Post-training quantization starts from a trained model and chooses its low-bit representation, sometimes using calibration inputs but no model-training gradient updates. GPTQ, AWQ and SmoothQuant are examples. Quantization-aware training exposes training to quantization effects, often with fake quantization: forward computations see rounded values while a surrogate gradient passes through the non-differentiable rounding step. The model can learn weights that tolerate the eventual grid.
Language-model The exponential of mean negative log-probability assigned to the observed next tokens in an evaluation corpus. is a useful first quality check: exp(−mean log p(token)). It measures whether the compressed model still assigns good probability to held-out text. Compare it on the same corpus, tokenizer, context policy and evaluation code. Small differences are meaningful only when the measurement is controlled.
Keep calibration and evaluation data separate. Calibration is part of choosing the compressed model: scales, clipping thresholds and compensation can all be fitted to it. Measuring only on those same tokens risks rewarding a representation that memorized their numerical quirks. A held-out set asks the more useful question: did the choice preserve behavior on inputs that did not help choose the codes?
It also helps to compare layers before comparing an entire generation. Feed the same saved activation matrix through the original layer and the quantized layer, then inspect output error per channel and per token. A single sensitive module may justify a wider dtype while the bulk of the model remains small. That is a measured mixed-precision decision: it spends extra bits where an experiment shows they help.
Perplexity averages over many predictions. A small average change can coexist with a broken code task, lost rare-language ability, worse long-context retrieval or different instruction following. Test the work you need the model to do. Also compare deterministic logits or teacher-forced losses when diagnosing drift: one changed sampled token can send two free-running generations down different paths and make a tiny initial numerical difference look enormous.
Smaller weights in a working server
Return to the 70-billion-weight model. BF16's 140 GB is easy to count because every weight costs two bytes. A homogeneous Q4_K payload costs 70 billion × 4.5/8 = 39.375 GB, before file metadata and any tensors that stay in a wider type. That is 4.375 GB more than the 35 GB four-bit headline. Over billions of values, a small fractional bit is a substantial allocation.
At a hypothetical sustained weight-read bandwidth of 2,000 GB/s, streaming 140 GB takes 70 ms. Streaming 39.375 GB takes about 19.7 ms. The corresponding weight-only ceilings are about 14.3 and 50.8 decoded tokens per second for one sequence. These weight-only bounds assume dense, batch-one decoding and enough memory to hold the model.
The calculator includes block overhead for the named types. It shows storage in both decimal GB and binary GiB and uses GB/s for bandwidth. Try Q4_0 and Q4_K: both use 4.5 bits per weight, despite distributing their metadata differently. Compare their reconstruction quality separately.
- Effective bits / weight
- Weight storage
- Weight-only ceiling
Dense model, batch 1, one complete weight stream per token. Excludes KV cache, unquantized tensors, alignment, communication and kernel overhead. NVFP4 counts one FP32 tensor scale for this illustrative tensor.
Keep compression across the expensive link
There is a tempting implementation mistake: load a four-bit checkpoint, expand the whole thing to FP16 in accelerator memory, then use an ordinary FP16 multiply. Loading from disk was cheaper, but serving now reads the wide weights again. To reduce runtime weight traffic, keep the persistent representation compressed and expand only a tile where the arithmetic needs it.
One optimized W4A16 path brings packed codes and scales from HBM into the kernel, stages tiles, unpacks weights into FP16 register fragments, and immediately consumes them in a matrix multiply. The expanded tile never becomes a full FP16 weight matrix in HBM. Marlin's CUDA kernel is a concrete example of fusing dequantization with FP16 tensor-core work.
The scale also has to reach the right fragment. Per-group scales introduce boundaries along the reduction dimension. Kernel tiling, packing and scale placement are designed together so the unpacking instructions can be interleaved with loads and matrix operations. A format that looks elegant as a table may still be expensive if each tile needs irregular loads or too many registers.
CPU kernels can multiply quantized weights by quantized activation blocks with integer dot products. FP8 and Blackwell block-scaled FP4 paths can use native low-bit arithmetic. Trace where your runtime expands or rescales the data to find the remaining bottleneck.
When smaller stops meaning faster
During prefill, many prompt positions reuse each weight. Larger decode batches also reuse the same weight across multiple sequences. As reuse grows, arithmetic becomes a larger part of the cost; the inference roofline shows that transition. A W4A16 kernel still feeds 16-bit arithmetic, so it may add unpacking work without raising the compute peak. It can lose to a well-tuned BF16 kernel for some shapes.
Weight-and-activation quantization can raise the available arithmetic throughput, provided the hardware has the matching instruction path and the kernel keeps it busy. Yet activation scaling, conversions, launches, small matrices and unsupported layers can consume the gain. A tensor-core peak describes a particular operation under favorable conditions; a server performs a mixture of operations.
Long contexts can shift pressure to KV-cache traffic. Splitting a model across accelerators adds communication. A four-bit model that fits one device may gain more by avoiding that communication than by the raw byte ratio. Conversely, if it still spills to host memory, device-memory bandwidth is the wrong bandwidth to put in a prediction. Quantization changes the budget, and the changed budget may change the whole deployment.
For a fair measurement, keep the model, prompts, context lengths and generation policy fixed. Report prefill tokens per second, decode tokens per second, batch size, total resident memory and quality. Include enough warm-up to get past compilation and loading. Measure the thing you intend to optimize: interactive latency for one user and aggregate throughput for a busy server can prefer different precision settings.
Compression helps serving when it survives into the memory traffic or arithmetic that limits the workload. Predict the gain from the bytes and operations on that path.
Two weights in a byte
At the machine layer, two four-bit codes share one eight-bit byte. The decoder needs an exact nibble convention. Suppose the first code is 3 and the second is 12. Put the first in the low nibble and the second in the high nibble: (12 << 4) | 3 gives 0xC3. Read them back with byte & 0x0F and byte >> 4.
Nibble order assigns tensor elements to each half of a byte; endianness orders the bytes of a multi-byte word. ggml Q4_0 puts code j in the low nibble and code j + 16 in the high nibble, for j from 0 to 15. A decoder that assumes consecutive pairs will return the right set of values in the wrong order.
The meaning of the nibble is another independent decision. A two's-complement signed nibble decodes 0xF as −1. An offset-binary code with zero-point 8 decodes it as 7. The lab uses offset binary: codes 0 through 15 represent integers −8 through 7 before scaling. Q4_0 uses that q − 8 reconstruction with its stored scale.
- Extract low code
- Build FP16
- Subtract FP16 1032
- Apply example scale 0.25
Offset-binary codes represent q − 8. This is not two's-complement INT4. The example byte stores consecutive values; ggml Q4_0 pairs element j with element j + 16.
Make the bit pattern do the conversion
FP16's bit pattern 0x6400 represents 1024. At that magnitude its spacing is exactly 1: the ten-bit fraction contributes an integer because 2¹⁰/2¹⁰ = 1. OR a code q from 0 to 15 into the low fraction bits and you get the floating-point value 1024 + q. Subtract 1024 to recover an unsigned code, or subtract 1032, whose FP16 pattern is 0x6408, to recover q − 8.
For q = 3, 0x6403 is FP16 1027. Subtract 1032 and the result is −5, exactly. With scale 0.25, the reconstructed weight is −1.25. The exponent makes each low fraction bit an integer place, so these steps are exact in FP16. Applying the scale may round.
Marlin's implementation works on two FP16 lanes packed into a 32-bit register. Its pinned source uses EX = 0x64006400 and SUB = 0x64086408. For nibbles still shifted left by four, it uses MUL = 0x2C002C00, two FP16 copies of 1/16, and ADD = 0xD480D480, two copies of −72. The fused expression (1024 + 16q) × (1/16) − 72 again gives q − 8. A bitwise LOP3 instruction combines masking and exponent insertion.
Decode a complete block
The following is a complete Q4_0 block: a little-endian FP16 scale of 0.25, followed by 16 packed bytes. The low nibbles encode the first 16 weights, the high nibbles the next 16. We choose the first half to run from −2 to 1.75 in steps of 0.25 and the second half in reverse order. Those codes produce the bytes shown below.
- d: FP16 scale0x3400 = 0.25 in little-endian order
- qs: 32 weight codesLow nibbles: 0…15. High nibbles: 15…0.
You can decode the displayed block with Python's standard library. The <e format reads a little-endian IEEE binary16 value. Keeping the two halves separate is essential: appending both nibbles of each byte consecutively would interleave the original elements incorrectly.
import struct
block = bytes.fromhex(
"00 34 F0 E1 D2 C3 B4 A5 96 87 78 69 5A 4B 3C 2D 1E 0F"
)
scale = struct.unpack_from("<e", block)[0]
packed = block[2:]
low = [scale * ((byte & 15) - 8) for byte in packed]
high = [scale * ((byte >> 4) - 8) for byte in packed]
weights = low + high
assert len(weights) == 32
assert weights[0] == -2.0
assert weights[16] == 1.75
print(weights)The ggml block definitions also give Q8_0 as a two-byte FP16 scale followed by 32 signed bytes: 34 bytes, or 8.5 bits per weight. Q4_0 is 18 bytes for 32 weights: 4.5 bits per weight. Static assertions in the source enforce both sizes.
To decode an INT4 tensor, read its type, block dimensions, field order, scale encoding, element order and zero-point rule. A kernel may repack that canonical representation into another order for efficient register loads. Repacking can preserve the exact numbers while changing every byte's position.
What fewer bits buys in silicon
The last question is why hardware designers care so much about a handful of bits. Start with an unsigned schoolbook multiplier. An n-bit by n-bit multiplication forms n² single-bit partial products before adding them together. Fewer significant bits mean fewer such products, shorter wires and smaller intermediate structures. Floating-point multipliers also spend hardware on signs, exponents, normalization and rounding.
As a first-order model, the significand multiplier's area and switched work grow roughly with the square of its width. That makes lower precision attractive for dense arrays of multiply-accumulate units. Total chip power also depends on accumulation width, circuit design, voltage, utilization and the process node.
Moving the operands can cost more than multiplying them. Mark Horowitz's 2014 ISSCC energy analysis uses illustrative 45 nm figures of about 3.7 pJ for a 32-bit floating-point multiply and about 640 pJ for a 32-bit DRAM access. These historical figures show how fetching data from far away can dominate a local arithmetic operation.
Quantization attacks both sides. Compact weights put more useful values into a memory transaction and into on-chip storage. Narrow arithmetic lets a fixed hardware budget execute more products. Shared scales spend a little metadata and conversion machinery to give those narrow operands a usable range. The inference dive explains tensor-core tiles and the memory hierarchy; the formats here are choices for the operands traveling through it.
H100 SXM · Hopper
B200 · Blackwell
Dense theoretical peaks, normalized separately to each GPU's BF16 peak. Structured sparsity is excluded. Real kernel and application gains depend on shapes, scaling and memory traffic.
For NVIDIA H100 SXM, Hopper, the architecture whitepaper lists about 989 TFLOP/s dense FP16/BF16 tensor throughput and 1,979 TFLOP/s dense FP8: a 2× arithmetic peak. Hopper does not have Blackwell's native FP4/NVFP4 tensor-core path. It can still run a weight-only INT4 kernel that expands weights for wider arithmetic.
For Blackwell B200, the HGX table gives per-GPU dense rates of 2.25 PFLOP/s BF16, 4.5 PFLOP/s FP8 and 9 PFLOP/s FP4, dividing the eight-GPU totals and removing the stated sparsity factor where needed: 2× and 4× relative to its own BF16 peak. NVIDIA's NVFP4 description explains the block scales supporting that low-bit path. The figure uses dense B200 rates per GPU; product variants and instruction modes have different peaks.
Accumulation deserves its own precision budget. A dot product can sum thousands of products with different signs and magnitudes. Even when individual operands have only four or eight bits, repeatedly rounding that running sum to the same tiny alphabet would lose information quickly. Wider accumulation keeps many small contributions alive until the result is rescaled or converted for the next operation. Low-bit inputs are therefore compatible with a substantially wider internal result.
Those ratios are available arithmetic capacity. They become application speedups only when enough of the workload uses those operations efficiently. A W4A16 kernel running FP16 multiplies uses the FP16 arithmetic path. A memory-limited decode step can benefit from smaller weights even when its multiply throughput stays unchanged.
Follow one weight back up
Our original 0.1372 belongs to a channel with a particular activation distribution, a group with a scale, a block with packed codes, and a kernel that must deliver the reconstructed value to an arithmetic unit. The “right” four-bit code depends on all of those surroundings.
At the top, quantization made a 70-billion-weight model plausible on smaller hardware. Underneath, it chose a smaller alphabet, adapted that alphabet to groups, protected difficult channels and sometimes used correlations to repair rounding. At the bottom, it saved memory transactions and supplied cheaper operands. Every layer is part of the same trade: preserve the model's useful computation while spending fewer bits to carry it out.
The advantage comes from fitting representation, quantization algorithm and kernel to the workload. Four bits is enough only when those four bits are spent well.
End of Quantization · 8 levels, Idea to Silicon
Where to next
Next in AI serving · 03 of 06
Long context, from a million tokens to a fixed-size state
A million tokens can hold a codebase. Following one question through them reveals three different limits: attention work, growing caches, and what a compressed state can remember.
Practical notes
Choose Q4_K_M, Q5_K_M or Q8_0 with a same-model test
Compare exact GGUF file sizes, benchmark all three with fixed settings, and reject candidates that fail a quality threshold. A Llama 3 8B example favors Q5_K_M under a six-GiB weight budget.