Count prompt tokens with the model’s tokenizer
Use tiktoken for a known OpenAI encoding or a Hugging Face checkpoint’s tokenizer files. Reserve output space and account for chat templates before checking context and cost.
Count tokens with the tokenizer for the exact model you will call. Character counts and whitespace splits are rough estimates: token vocabularies, merge rules, special tokens and chat templates differ. A plain string’s token count is only one part of a structured chat request.
Count a string with tiktoken
Install tiktoken in a virtual environment, then run this example. cl100k_base is an explicit encoding choice for demonstration, not a claim that every current OpenAI model uses it.
python3 -m venv /tmp/token-example-env
/tmp/token-example-env/bin/pip install tiktoken tokenizers transformers==5.23.0 jinja2
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, world!"
ids = encoding.encode(text)
print(ids)
print("tokens:", len(ids))
assert encoding.decode(ids) == text
Output:
[9906, 11, 1917, 0]
tokens: 4
When the library knows your actual model name, tiktoken.encoding_for_model(model_name) selects its encoding. If it raises an unknown-model error, resolve the model’s documented tokenizer instead of silently substituting another. See the official tiktoken documentation.
Count with Hugging Face tokenizers
For a compatible checkpoint, obtain its tokenizer.json and use Tokenizer.from_file("tokenizer.json"). Follow the checkpoint’s special-token policy: automatically adding BOS or EOS can change the count. The Tokenizers API defines both file loading and encode.
This self-contained small-vocabulary example needs no model download. It illustrates the API, not a production model’s segmentation:
from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.pre_tokenizers import Whitespace
vocab = {"[UNK]": 0, "Hello": 1, ",": 2, "world": 3, "!": 4}
tokenizer = Tokenizer(WordLevel(vocab, unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
encoded = tokenizer.encode("Hello, world!")
print(encoded.tokens)
print(encoded.ids)
print("tokens:", len(encoded.ids))
Output:
['Hello', ',', 'world', '!']
[1, 2, 3, 4]
tokens: 4
These two encoders happen to give four tokens for this sentence, but assign different IDs. Try Hello worlds!: the toy vocabulary produces an unknown token, whereas a production subword tokenizer can represent unfamiliar words using smaller units. Never send the toy IDs to a real model.
Include the rendered chat prompt
Transformers 5.23.0 returns a BatchEncoding by default from apply_chat_template. Count result["input_ids"], not len(result): the latter counts mapping fields. Alternatively, pass return_dict=False to get a list of token IDs for a single conversation. Jinja2 renders the checkpoint’s chat template; install it alongside Transformers as above.
Save this as chat_tokens.py. The pinned Qwen2.5-7B-Instruct tokenizer downloads only tokenizer/configuration files, not the model weights:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"Qwen/Qwen2.5-7B-Instruct",
revision="a09a35458c702b33eeacc393d103063234e8bc28",
)
messages = [{"role": "user", "content": "Hello, world!"}]
result = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_dict=True,
)
print("message tokens:", len(tokenizer.encode("Hello, world!", add_special_tokens=False)))
print("chat tokens:", len(result["input_ids"]))
ids = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_dict=False,
)
assert ids == result["input_ids"]
Run it with the optional model-backend advisory disabled; tokenizer-only work does not require PyTorch:
TRANSFORMERS_NO_ADVISORY_WARNINGS=1 /tmp/token-example-env/bin/python chat_tokens.py
Output:
message tokens: 4
chat tokens: 33
The extra tokens include Qwen’s default system message, role separators and the assistant-generation prefix. A different chat history or checkpoint template changes the count. Tool schemas and tool-call histories also add input; multimodal requests need model-specific accounting beyond this text example. The official chat-template guide and tokenizer API explain the rendering and return types.
For a hosted API, use its reported usage to reconcile your estimate. Framing rules may be provider-specific, and a tokenizer library’s string encoder does not necessarily reproduce the whole request envelope.
Reserve context and estimate cost
Keep input tokens plus the output reservation within the model’s documented context capacity, and obey its separate output limit. Here is illustrative arithmetic with made-up prices in currency units per million tokens:
input_tokens, output_budget, context_limit = 6000, 2000, 16384
input_rate, output_rate = 0.50, 1.50
assert input_tokens + output_budget <= context_limit
maximum_cost = (input_tokens * input_rate + output_budget * output_rate) / 1_000_000
print(f"budgeted cost: {maximum_cost:.6f}")
print("context headroom:", context_limit - input_tokens - output_budget)
The output is a budgeted cost of 0.006000 and headroom of 8384 tokens. Actual billing depends on generated output, cache treatment and the provider’s pricing rules; substitute the current rates for your deployment.
Go deeper into tokenization during inference.