Topics / ai-serving

AI serving

What it takes to run a model once it is trained: generating tokens fast, fitting weights into memory, coping with long prompts, and connecting models to data and tools.

Dives
6
Levels
51
Notes
8
Reading
4.8h
Start with LLM inference
How to read thisEvery line is a diveEach tick is one level, drawn at the layer of the stack it explains. Longer lines go further down, from why a thing exists to the silicon that runs it.6 dives · 51 levels

The path

Read in this order and each dive leans on the ones before it. Every dive also stands on its own.

  1. 01

    How an LLM writes the next word

    Follow one prompt through a real model: tokenizer, embeddings, attention, sampling and the KV cache, then down to why serving is bound by memory bandwidth and how tensor cores do the actual work.

    Idea to Silicon

    55 min9 levels

  2. 02

    Quantization, from 16 bits to 4

    Squeeze a 70-billion-weight model into fewer bytes. Explore the floating-point encodings, tame outliers, compensate rounding errors, then unpack four-bit weights inside a GPU kernel.

    Idea to Silicon

    40 min8 levels

  3. 03

    Long context, from a million tokens to a fixed-size state

    A million tokens can hold a codebase. Following one question through them reveals three different limits: attention work, growing caches, and what a compressed state can remember.

    Idea to Machine

    42 min9 levels

  4. 04

    How a GPU runs a kernel, from threadIdx to DRAM

    Follow a CUDA thread into a warp and onto an SM. Make branches diverge, scatter a memory request across sectors, and fix a transpose one bank conflict at a time.

    Idea to Silicon

    42 min9 levels

  5. 05

    Vector databases, from a question to a dot product

    Follow one search for a laptop that will not charge: learned geometry, graph navigation, compressed codes, filtered retrieval, and the memory traffic behind every similarity score.

    Idea to Machine

    45 min9 levels

  6. 06

    MCP, from a tool call to bytes on a pipe

    Follow one issue-tracker lookup through a model, a host, JSON-RPC, authorization and the kernel. Build messages, break a stream and inspect the bytes, with the 2026 and 2025 protocols kept apart.

    Idea to Machine

    38 min7 levels

Notes

Practical answers in ai serving.

  1. Note3 min read

    Write a minimal MCP server with the TypeScript v2 SDK

    Register a schema-validated tool with @modelcontextprotocol/server and connect it over stdio. Use MCP Inspector to list and call the tool.

  2. Note3 min read

    Transcribe audio locally with word timestamps

    Use faster-whisper with word_timestamps and VAD, consume its segment generator, and preserve source-time offsets. Choose model size and compute type for your machine.

  3. Note3 min read

    Measure LLM tokens per second without mixing workloads

    Report prefill throughput, decode throughput and time to first token separately. A bandwidth estimate gives an optimistic bound for small-batch dense decoding.

  4. Note3 min read

    Tune pgvector HNSW for recall and filtered queries

    Start with default graph settings, tune ef_search against exact results, and enable iterative scans for filtered queries. Build memory and graph construction settings solve different problems.

  5. Note3 min read

    Choose cosine, dot product or L2 for embeddings

    For unit-normalized embeddings, cosine similarity, dot product and squared L2 produce the same ranking. Match your index operator and ordering to the embedding model.

  6. Note3 min read

    Choose Q4_K_M, Q5_K_M or Q8_0 with a same-model test

    Compare exact GGUF file sizes, benchmark all three with fixed settings, and reject candidates that fail a quality threshold. A Llama 3 8B example favors Q5_K_M under a six-GiB weight budget.

  7. Note3 min read

    Estimate KV-cache memory before loading a model

    Compute GQA cache bytes from layer count, KV heads, head dimension and retained tokens. Use a separate latent-cache calculation for MLA models.

  8. Note3 min read

    Count prompt tokens with the model’s tokenizer

    Use tiktoken for a known OpenAI encoding or a Hugging Face checkpoint’s tokenizer files. Reserve output space and account for chat templates before checking context and cost.