Measure LLM tokens per second without mixing workloads
Report prefill throughput, decode throughput and time to first token separately. A bandwidth estimate gives an optimistic bound for small-batch dense decoding.
Measure prompt processing and generation separately. Prefill throughput counts input tokens processed per second. Decode throughput counts newly generated tokens per second. Time to first token, or TTFT, measures the delay from request submission to the first streamed token, including queueing and prompt work. One tokens-per-second number cannot represent all three.
Benchmark the local engine
From a built llama.cpp checkout, point -m at an existing GGUF checkpoint:
./build/bin/llama-bench -m model.gguf -p 512 -n 128 -r 5 -o json
./build/bin/llama-bench -m model.gguf -p 0 -n 128 -d 4096 -r 5 -o json
The llama-bench documentation at revision fc504f91d8cf defines -p as prompt length, -n as generation length, -d as prefilled context depth and -r as repetitions. The first invocation runs separate prompt-processing and token-generation tests. The second disables the prompt test and measures generation at a populated cache depth. Its -d tokens establish the initial context rather than counting as generated output.
Do not label the first command “128-token generation after a 512-token prompt.” Separate pp512 and tg128 rows describe separate tests; -pg 512,128 selects the combined workload. A -b setting controls token batching inside the engine, not a count of simultaneous end users.
Record the binary revision, model hash, quantization, thread count, accelerator backend, GPU placement, cache dtype and context depth with the result. Keep warm-up enabled and inspect the repetition spread. Compare the same settings on the same power and thermal conditions.
Estimate a bandwidth ceiling
For dense, small-batch decoding with weights resident in device memory, a first approximation is one read of the active weights per decode step:
seconds per token ≈ bytes moved per token / sustained bytes per second.
This Python calculator uses decimal GB, not GiB. Its bandwidth and utilization inputs are illustrative assumptions, not specifications for a particular GPU:
parameters = 8_000_000_000
bits_per_weight = 4.5
bandwidth_gb_s = 400
utilization = 0.70
weight_bytes = parameters * bits_per_weight / 8
seconds_per_token = weight_bytes / (bandwidth_gb_s * 1e9 * utilization)
print(f"weight bytes: {weight_bytes / 1e9:.2f} GB")
print(f"optimistic decode: {1 / seconds_per_token:.1f} tokens/s")
Output:
weight bytes: 4.50 GB
optimistic decode: 62.2 tokens/s
Add KV reads, cache writes, activation traffic and metadata for a fuller traffic estimate. Kernel launch overhead, compute and transfers can make real throughput lower. Expert models need an active-weight model, and repeatedly used weights may receive cache reuse; this simple estimate is not a universal hardware limit.
Separate engine speed from service latency
A local microbenchmark omits the scheduler, HTTP transport and competing requests. For a service, timestamp request submission, the first actual output token, subsequent tokens and completion. Exclude empty streaming chunks and protocol metadata. State whether decode throughput excludes the first token; one common inter-token estimate is (output_tokens - 1) / (last_token_time - first_token_time).
Batching can improve aggregate output tokens per second while worsening individual TTFT or inter-token latency. Report both per-request latency distributions and total throughput at a stated concurrency and input/output length distribution. Prefix-cache hits should be a separate workload from cold prompts.
Go deeper into inference serving and GPU throughput.