Topics / ai-serving
AI serving
What it takes to run a model once it is trained: generating tokens fast, fitting weights into memory, coping with long prompts, and connecting models to data and tools.
- How an LLM writes the next word, 9 levels · 55 min · Idea to Silicon
- Quantization, from 16 bits to 4, 8 levels · 40 min · Idea to Silicon
- Long context, from a million tokens to a fixed-size state, 9 levels · 42 min · Idea to Machine
- How a GPU runs a kernel, from threadIdx to DRAM, 9 levels · 42 min · Idea to Silicon
- Vector databases, from a question to a dot product, 9 levels · 45 min · Idea to Machine
- MCP, from a tool call to bytes on a pipe, 7 levels · 38 min · Idea to Machine
The path
Read in this order and each dive leans on the ones before it. Every dive also stands on its own.
- 01
How an LLM writes the next word
Follow one prompt through a real model: tokenizer, embeddings, attention, sampling and the KV cache, then down to why serving is bound by memory bandwidth and how tensor cores do the actual work.
Idea to Silicon55 min9 levels
- 02
Quantization, from 16 bits to 4
Squeeze a 70-billion-weight model into fewer bytes. Explore the floating-point encodings, tame outliers, compensate rounding errors, then unpack four-bit weights inside a GPU kernel.
Idea to Silicon40 min8 levels
- 03
Long context, from a million tokens to a fixed-size state
A million tokens can hold a codebase. Following one question through them reveals three different limits: attention work, growing caches, and what a compressed state can remember.
Idea to Machine42 min9 levels
- 04
How a GPU runs a kernel, from threadIdx to DRAM
Follow a CUDA thread into a warp and onto an SM. Make branches diverge, scatter a memory request across sectors, and fix a transpose one bank conflict at a time.
Idea to Silicon42 min9 levels
- 05
Vector databases, from a question to a dot product
Follow one search for a laptop that will not charge: learned geometry, graph navigation, compressed codes, filtered retrieval, and the memory traffic behind every similarity score.
Idea to Machine45 min9 levels
- 06
MCP, from a tool call to bytes on a pipe
Follow one issue-tracker lookup through a model, a host, JSON-RPC, authorization and the kernel. Build messages, break a stream and inspect the bytes, with the 2026 and 2025 protocols kept apart.
Idea to Machine38 min7 levels
Notes
Practical answers in ai serving.
Write a minimal MCP server with the TypeScript v2 SDK
Register a schema-validated tool with @modelcontextprotocol/server and connect it over stdio. Use MCP Inspector to list and call the tool.
Transcribe audio locally with word timestamps
Use faster-whisper with word_timestamps and VAD, consume its segment generator, and preserve source-time offsets. Choose model size and compute type for your machine.
Measure LLM tokens per second without mixing workloads
Report prefill throughput, decode throughput and time to first token separately. A bandwidth estimate gives an optimistic bound for small-batch dense decoding.
Tune pgvector HNSW for recall and filtered queries
Start with default graph settings, tune ef_search against exact results, and enable iterative scans for filtered queries. Build memory and graph construction settings solve different problems.
Choose cosine, dot product or L2 for embeddings
For unit-normalized embeddings, cosine similarity, dot product and squared L2 produce the same ranking. Match your index operator and ordering to the embedding model.
Choose Q4_K_M, Q5_K_M or Q8_0 with a same-model test
Compare exact GGUF file sizes, benchmark all three with fixed settings, and reject candidates that fail a quality threshold. A Llama 3 8B example favors Q5_K_M under a six-GiB weight budget.
Estimate KV-cache memory before loading a model
Compute GQA cache bytes from layer count, KV heads, head dimension and retained tokens. Use a separate latent-cache calculation for MLA models.
Count prompt tokens with the model’s tokenizer
Use tiktoken for a known OpenAI encoding or a Hugging Face checkpoint’s tokenizer files. Reserve output space and account for chat templates before checking context and cost.
Other topics
Systems and hardware
The machine under all the software: how a processor executes instructions, what memory really is, and how a GPU keeps thousands of threads busy.
6 dives · 4 notes · 4.6 h of reading
AI models
How neural networks learn from data and get shaped into assistants, and the architectures that turn text, images and sound into predictions.
6 dives · 1 note · 4.1 h of reading
Web and networking
Everything between a URL and a pixel: the protocols that carry bytes across the internet and keep them private and fresh, and the browser machinery that turns them into a page.
12 dives · 10 notes · 8.8 h of reading
Languages and runtimes
The work a language runtime does on your behalf: scheduling goroutines, collecting garbage, and running portable bytecode safely and fast.
6 dives · 5 notes · 4.5 h of reading
Infrastructure and operations
Deploying and operating software: Kubernetes workloads, Helm releases, Ansible automation, Raspberry Pi clusters and the practical checks that keep systems running.
6 notes · 0.3 h of reading
Application development
Building applications and data workflows: framework patterns, Django and Wagtail models, Next.js integration, identity schemas and practical data scripts.
5 notes · 0.2 h of reading