Notes

Choose Q4_K_M, Q5_K_M or Q8_0 with a same-model test

Compare exact GGUF file sizes, benchmark all three with fixed settings, and reject candidates that fail a quality threshold. A Llama 3 8B example favors Q5_K_M under a six-GiB weight budget.

Choose the smallest quantization that fits your memory budget and passes a quality gate, then compare its speed on your machine. Use the same base checkpoint throughout. This worked procedure compares Meta-Llama-3-8B base weights in Q4_K_M, Q5_K_M and Q8_0; it combines published file sizes and quality measurements with commands for your own timing and acceptance test.

Set a budget and collect the files

Suppose your weight-file budget is 6 GiB, with additional memory reserved separately for the KV cache and runtime. Download these three files from QuantFactory/Meta-Llama-3-8B-GGUF at revision 1ca85c857dce. The repository’s file metadata supplies actual bytes, rather than an estimate from nominal bit width:

Filename File bytes GiB
Meta-Llama-3-8B.Q4_K_M.gguf 4,921,246,944 4.583
Meta-Llama-3-8B.Q5_K_M.gguf 5,733,500,128 5.340
Meta-Llama-3-8B.Q8_0.gguf 8,541,283,552 7.955

Q8_0 exceeds this weight budget. Keep it as a quality reference if your evaluation machine has sufficient memory. File size is not peak resident memory: verify actual allocation at your intended context length before deployment.

From the directory containing the files, record their hashes and sizes:

sha256sum Meta-Llama-3-8B.*.gguf
wc -c Meta-Llama-3-8B.*.gguf

Hold the benchmark settings fixed

Use a llama.cpp build from revision fc504f91d8cf. From that checkout, with the files in its working directory:

for quant in Q4_K_M Q5_K_M Q8_0; do
  ./build/bin/llama-bench -m "Meta-Llama-3-8B.${quant}.gguf"     -p 512 -n 128 -t 8 -ngl 0 -b 512 -ub 512     -ctk f16 -ctv f16 -fa off -r 5 -o json     > "bench-${quant}.json"
done

This is a CPU baseline: eight threads, no GPU layers, FP16 cache, Flash Attention off and five repetitions. Compare prompt-processing and generation rows separately; these are separate tests, not generation after the measured prompt. Keep warm-up enabled. For GPU deployment, change placement consistently for every candidate and repeat on the target machine. Published perplexity results below do not establish your throughput.

Apply a quality gate

For this example, accept at most 0.10 additional perplexity versus Q8_0. The llama.cpp Llama 3 8B scoreboard reports these no-importance-matrix results, produced at revision f364eb6f on an RTX 4090:

Quantization Published perplexity Increase versus Q8_0
Q4_K_M 6.407115 0.172831
Q5_K_M 6.288607 0.054323
Q8_0 6.234284 0

Those measurements describe the same base model and quantization recipes, but are not measurements of the exact QuantFactory artifacts above. Use them to choose a candidate, then evaluate your downloaded files. Perplexity is a next-token-prediction check, not a guarantee of task accuracy.

Put a fixed, held-out text corpus in eval.txt, large enough for 20 chunks of 512 tokens, and run the same test for each file:

for quant in Q4_K_M Q5_K_M Q8_0; do
  ./build/bin/llama-perplexity -m "Meta-Llama-3-8B.${quant}.gguf"     -f eval.txt -c 512 -b 512 -t 8 -ngl 0 --chunks 20     > "ppl-${quant}.log" 2>&1
done

Use identical corpus bytes and settings, and retain the reported uncertainty. A short corpus cannot reliably separate tiny changes. For an application, add representative tasks with explicit pass criteria; preserve the prompt template and sampling settings across formats.

Make the choice

I would select Q5_K_M as the candidate for this budget and quality gate: the published increase is below 0.10, Q4_K_M’s is above it, and Q8_0 exceeds six GiB. Q5_K_M saves 2,807,783,424 file bytes versus Q8_0 while costing 812,253,184 more than Q4_K_M. Deploy it only after the exact-file quality run, task checks and target-machine timing pass. If it fails, revisit the budget or model rather than assuming another four-bit recipe will behave identically.

For GGUF recipes, AWQ/GPTQ, FP8 and FP4 internals, use the quantization dive.