AI modelsDive 04 of 06

Vision models, from pixels to an answer

A receipt says $12.50. Follow its pixels through tensors, patches, attention and a language model to see how an image becomes evidence for an answer.

Levels
9
Reading
38 min
Depth
Idea to Machine
Published
Your question

What is the total on this receipt?

128 × 192 pixels8 × 12 patches96 toy vectors

Four bars per patch encode mean RGB and red-channel contrast.

Level 01/09Idea

One photo, one question

You upload a photo of a café receipt and ask, What is the total? The answer you want is $12.50. The receipt also contains $4.50, $6.00, $10.50 and $2.00. Reading a number is only part of the job. The model must find the right document, recognize the printing, associate the amount with the word total, and put that evidence into a sentence.

A A model that conditions language processing or generation on visual input, often by joining an image encoder to a language model., or VLM, turns the photo into numerical features that a language model can use. The transformer has no built-in concept of red, paper, money or a photograph. It accepts arrays, multiplies them by learned matrices, exchanges information between positions, and produces another array. Training makes those transformations useful for answering your question.

The image takes a different entrance from the question. Text is split into vocabulary items whose IDs select learned embedding vectors. An image usually becomes many vectors computed from regions of its pixels. Those visual vectors can have the same width as text embeddings without being entries in the text vocabulary. There need not be a word that decodes from any one of them.

That distinction explains why the photo costs context. If a connector passes 576 image vectors into the decoder, those vectors occupy 576 sequence positions. They can consume more of the input than your short question. A different connector might compress the image to a fixed set of queries; another might preserve thousands of visual positions. The upload size in megabytes does not determine that count. The image processor, vision encoder and connector do.

Three jobs, three places to lose the answer

Think of the request as three connected jobs. First preserve the evidence: the digits and decimal point must survive image preprocessing. Then represent it: the visual network must produce features that distinguish those marks and their layout. Finally use it: the language model must ground its answer in the features rather than complete a plausible café story from memory.

JobReceipt evidenceA failure you can recognize
PreserveThe lower line and its decimal point reach the encoderA centre crop cuts off the total
RepresentMarks become distinguishable features with positionThe small 2 becomes indistinguishable from a 7
UseThe answer selects the amount next to “total”The model confidently reports the subtotal

The route down follows those dependencies. We begin with decoded bytes, compare a local convolution with a patch transformer, look at how image-text training teaches useful features, then connect those features to a decoder. At the bottom are matrix multiplies, attention work and memory allocations. A photo does not bypass the cost of language inference; it creates more numerical input for it.

The sample throughout the figures is a small, procedurally drawn receipt: coffee $4.50, toast $6.00, subtotal $10.50, tax $2.00, total $12.50. Its pixelated lettering makes the arithmetic inspectable. A phone photograph adds perspective, blur, shadows and compression, but enters the same path after decoding. You can inspect every sample pixel locally; none of the controls sends an image to a service.

Keep two moments separate as you descend. During training, an objective and many examples change the model's weights. During inference, your receipt changes activations while those weights stay fixed. A successful answer can feel like the model has just learned to read your particular receipt. What happened is that its previously learned function was evaluated on new evidence.

Seeing means turning spatial evidence into features that can influence an answer. Preserving the pixels, representing their meaning and using that meaning are separate engineering problems.

Level 02/09Interface

A photograph becomes a tensor

The upload begins as a file, perhaps JPEG, PNG or HEIC. Its bytes are not a flat list of the red values in the picture. A decoder interprets the container and compression, reconstructs a pixel array, and applies the pipeline's orientation and colour handling. A rotated receipt must become upright before a later crop has any useful meaning. Transparency also needs a policy: an RGB model expects three channels, so an alpha channel must be removed or composited deliberately.

For a simple working representation, use 8-bit sRGB: three unsigned integers per pixel, each from 0 to 255. Our 128 × 192 sample contains 24,576 pixels and 73,728 RGB bytes before storage overhead. In an ordinary interleaved array, one row begins R, G, B, R, G, B. The next row follows it. Browser canvas readback adds an alpha byte, giving RGBA; the model input below uses only RGB.

sRGB numbers encode brightness nonlinearly. A channel value of 128 is not half the physical light of 255. A pretrained processor may work directly on these encoded channel values; do not add a linear-light conversion merely because it sounds more physically correct. Image colour space and metadata have explicit meanings in the PNG specification. The checkpoint's preprocessing contract decides which representation its weights expect.

Resize is an information decision

Suppose the original photo is 640 × 960. A fixed-size encoder might resize the shorter side to 336, producing 336 × 504, then take a central 336 × 336 crop. That removes 84 resized rows at each end. A total near the bottom can disappear before the network starts. Stretching to 336 × 336 keeps the whole page but changes the shapes of the letters. Fitting inside a square and padding keeps shape and coverage but spends pixels on blank borders.

Interpolation introduces another choice. Nearest-neighbour sampling chooses existing pixels; bilinear and bicubic methods combine neighbourhoods. Downsampling needs to suppress detail too fine for the new grid, otherwise it can alias into false patterns. A one-pixel decimal point can fade into the surrounding paper. More elaborate networks cannot recover evidence that preprocessing never delivered, though they may guess a likely value.

Next convert integers to floating point and divide by 255. Then apply the checkpoint's channel-specific normalization. The original OpenAI CLIP processor uses the mean and standard deviation shown in the lab, after bicubic resize, centre crop and RGB conversion. Each normalized value is the encoded channel intensity minus its channel mean, divided by its channel standard deviation.

xc,y,x = (bytey,x,c / 255 − μc) / σc

Built-in receipt

Preparing the receipt…

CLIP normalization constants. The built-in raster uses sRGB byte values; no model weights are loaded.

Select a pixel and inspect its actual RGB bytes, normalized values and channels-first tensor offsets. Coordinates start at the top left.

Negative values are expected: they mean below the channel mean, not negative light. The mean and standard deviation are fixed parameters of the preprocessing recipe, not statistics recomputed from this receipt. Replacing them with the receipt's own mean and deviation changes the input distribution. Likewise, accidentally normalizing twice turns a valid tensor into a different numerical object.

The familiar tensor shape is [batch, channels, height, width], abbreviated NCHW. For our one-image sample it is [1, 3, 192, 128]. With a contiguous NCHW layout, all red samples form one plane, followed by green and blue. In the following example the permutation changes axes; the framework can later choose a contiguous layout suited to its kernels. Shape describes the logical axes, while strides describe how those axes map to addresses.

After decoding and the chosen resize/crop
# rgb: decoded uint8 RGB array, shape [H, W, 3]
x = torch.from_numpy(rgb.copy()).permute(2, 0, 1).float() / 255
mean = x.new_tensor([0.48145466, 0.4578275, 0.40821073])
std = x.new_tensor([0.26862954, 0.26130258, 0.27577711])
x = (x - mean[:, None, None]) / std[:, None, None]
x = x.unsqueeze(0)  # [1, 3, H, W]

Batching adds a leading dimension, not an extra colour channel. A float32 version of this sample needs 294,912 bytes for its three planes, four times the RGB byte array. BF16 storage would take half that float32 space, with fewer significant bits. The network may use different precision in different operations; converting the input alone does not specify the precision of the whole model.

Check yourselfThe receipt's total was cut off by a centre crop. Which change restores evidence for the model?

Level 03/09Runtime

Start with the neighbours

We now have numbers, but adjacent pixels are still just separate samples. A useful first operation asks a local question: how does this pixel differ from its neighbours? A A learned local filter reused at every spatial position, usually combining several input channels into each output channel. slides a small weight array across the image. At each position, multiply corresponding values and add them. With a 3 × 3 filter on one intensity channel, that is nine products for each output.

The operation called convolution in most neural-network libraries is technically cross-correlation: the kernel is used in its stored orientation rather than flipped. Learned filters work either way if the convention is consistent. For a hand-written edge detector, the convention determines the sign. The lab uses cross-correlation and clamps out-of-bounds coordinates to the closest border pixel, so you can account for every term even in a corner.

Choose the vertical-edge kernel. It subtracts intensities on the left from intensities on the right, weighting the middle row twice. A uniform paper patch cancels to zero. A transition from dark ink to bright paper yields a positive response; the reverse yields a negative response. The displayed edge image takes the absolute value so both transitions appear bright. The readout preserves the signed number.

3 × 3 kernel

Built-in receipt

Filtered intensity

Preparing the receipt…

Cross-correlation on mean RGB intensity in [0, 1], replicated borders. Edges display |response|, clipped to [0, 1].

Apply a real 3 × 3 filter to the same receipt. Compare edge detection, a nine-sample average and identity; use the coordinates to inspect a particular response.

The blur kernel gives every neighbour weight 1/9. Its output is a local average, which smooths paper texture and softens the printing. The identity kernel keeps only the centre sample. These are three different operations with exactly the same multiply-and-sum structure. A trained network discovers many useful filters rather than choosing from this short menu.

Locality buys a useful bias

A single 3 × 3 layer can only depend on a 3 × 3 The input positions that can influence a given output position through the layers seen so far.. Stack two stride-one layers and the receptive field grows to 5 × 5; stack three and it grows to 7 × 7. This counts possible dependencies. Actual influence depends on the learned weights and activations. Strides, dilation and pooling change the growth rule.

Sharing a kernel across positions means that a detector for the vertical stroke of a 1 is also useful when the 1 moves across the receipt. For an infinite image and stride one, translating the input translates the output. This property is translation equivariance. Padding boundaries and strided downsampling limit exact equivariance in practical networks. Equivariance also differs from invariance: keeping the same classification after moving an object is a property of the whole system, often encouraged by pooling and data augmentation.

These are inductive biases: assumptions built into the architecture before learning. Local connections assume nearby samples are useful together. Shared weights assume a useful local detector should work in many places. Neither assumption says that position is irrelevant. For our receipt, the same digit pattern can occur in the subtotal and the total; layout still decides which amount answers the question.

A real convolutional layer usually has multiple input and output channels. A filter for one output channel has a separate 3 × 3 slice for every input channel, and sums all those products. With 3 input channels and 64 output channels, a 3 × 3 layer has 3 × 3 × 3 × 64 = 1,728 weights, plus 64 biases if enabled. The weight count does not grow with image width or height. Reuse is the point.

As layers accumulate, local edges can support strokes, strokes can support digit-like patterns, and wider receptive fields can combine those patterns. That is an explanation of what the architecture permits, not a guarantee that each layer learns one clean human category. Learned feature channels usually mix several properties.

Why move to a transformer? Global relationships can be reached directly through attention rather than only through a chain of expanding local neighbourhoods. That helps represent associations such as this number belongs to that label. It also removes some helpful constraints, increasing the burden on data and training. The original Vision Transformer paper demonstrated that processing patch sequences can work extremely well with sufficient pretraining data. The next operation makes the receipt into such a sequence.

Convolutions reuse a local rule everywhere. Transformers let positions exchange information more freely, after an image-to-sequence operation establishes the positions.

Level 04/09Runtime

Cut the image into a sequence

A Vision Transformer, or ViT, divides an input image into non-overlapping squares. With a 224 × 224 input and 16 × 16 patches, there are 14 patches across and 14 down: 196 spatial positions. Each RGB patch contains 16 × 16 × 3 = 768 values. Flattening those values gives a vector, then a learned linear projection turns that vector into the encoder's chosen feature width.

For patch p, write its flattened samples as xp. With a projection matrix E of shape [768, D], the patch embedding is xpE + b, a D-dimensional vector. D is a design choice. The input length happens to equal 768 for a 16-pixel RGB patch, but the output width need not equal that length. A 14-pixel RGB patch has 588 inputs and can still project to 1,024 features.

Flattening itself loses no samples. It chooses an ordering for them. The subsequent projection may discard information, depending on its dimensions and weights, and later layers transform it further. A patch boundary does not establish an object boundary: a decimal point can straddle two patches, and one patch can contain parts of several letters. The learned representation must work with these arbitrary cuts.

Square input resolution
Patch side length

Square stretch for this patch demonstration. The four displayed projection coordinates use fixed toy weights, not a trained ViT.

Change resolution and patch size, then inspect a projected patch. The grid count is exact; the displayed four-coordinate projection is computed from the sample with toy weights.

Changing from 224/16 to 336/14 gives 24 × 24 = 576 spatial tokens. Both the pixels per side and the patches per side changed. If you double both image dimensions while keeping the patch size, the number of spatial tokens grows fourfold. This is why quoting only an image's longest side is insufficient to estimate its transformer cost.

A common implementation expresses patch embedding as a convolution whose kernel and stride both equal the patch size. For each output location, it multiplies the same learned weights by the local RGB patch. Because the stride skips a full patch, neighbouring output locations use disjoint inputs. There is no separate algorithmic magic in spelling it as a convolution rather than as patch extraction followed by a matrix multiply.

Patch embedding as a strided convolution
# x: [batch, 3, height, width]; dimensions divisible by p
patch_embed = torch.nn.Conv2d(3, d, kernel_size=p, stride=p)
z = patch_embed(x)       # [batch, d, height/p, width/p]
z = z.flatten(2)         # [batch, d, number_of_patches]
z = z.transpose(1, 2)    # [batch, number_of_patches, d]

A sequence needs a map back to the page

The usual row-major ordering lists patches left to right across the top row, then proceeds downwards. Index 0 is the top left, and index 23 is the right edge of the first row in a 24-column grid. Index 24 starts the next row. A sequence index is useful bookkeeping, but attention also needs a signal that expresses spatial arrangement.

One method adds a learned positional vector to each patch embedding. The original ViT uses a learned one-dimensional position table over the ordered grid. When the grid changes, implementations can interpolate the spatial position table. Another method constructs position features explicitly from horizontal and vertical coordinates. These choices tell the network that nearby columns and rows have relationships that two unrelated slots in a bag do not.

A row-major sequence preserves a mapping to two-dimensional coordinates. Two adjacent sequence indices can cross a row boundary, while vertical neighbours are one grid-width apart.

Rotary position embedding, RoPE, encodes position by rotating pairs of query and key coordinates inside attention. In a two-dimensional version, subsets of those coordinates can carry horizontal and vertical position. The dot product then depends on relative displacement along those axes. This gives a way to express layout without adding one fixed lookup vector to every patch. A specific architecture determines which dimensions rotate and how; “2D RoPE” names a family of position schemes rather than one universal file format.

The encoder also needs a way to produce an image-level representation when the task is classification or retrieval. A [CLS] token is an extra learned input vector that can gather information through attention. It is not another patch. Alternatively, pool the patch outputs, for example by averaging them, then project that aggregate. Pooling avoids a dedicated summary position, but the final summary still compresses spatial detail.

For receipt reading, the spatial outputs matter. A single whole-image vector can indicate “receipt” while leaving little recoverable detail about the last amount. A connector can therefore take the patch outputs and discard the encoder's [CLS] output. The CLIP ViT implementation shows the strided convolution, class embedding and learned position embedding; which outputs a VLM selects is a separate connector decision.

Check yourselfA 336 × 336 image uses 14 × 14 patches. How many spatial patch tokens exist before any connector compression?

Level 05/09Runtime

Let the patches talk

Initially, each patch embedding depends on its own samples and position. The amount $12.50 and its label can occupy different patches. Self-attention lets every patch update its representation using information from other positions. In a conventional full-attention image encoder there is no causal rule that hides patches to the right or below: the whole image is available together.

For each attention head, learned matrices turn the input features into queries Q, keys K and values V. A query at position i scores a key at position j using a dot product. Divide by the square root of the head's key width dk, take a softmax over all j, and use the resulting positive weights to combine the value vectors.

aij = exp(qi · kj / √dk) / Σm exp(qi · km / √dk)
oi = Σj aijvj

The normalization makes each row sum to one. Scaling by √dk helps control how the scores grow as the head width increases. These are weights for mixing values, not calibrated probabilities that a patch contains the answer. A patch with a high weight can contribute an unhelpful value, or a value used to suppress a feature later.

The lab creates an 8 × 12 grid from our original receipt. It measures each patch's mean red-channel darkness, then builds two small attention heads. The ink head scores patches by the product of their darkness values. The row head favours nearby vertical positions. These are deliberately specified features, so you can follow why the map changes rather than treating an unknown model's colours as an explanation.

Toy attention head

Weights over 8 × 12 patches

Largest attention weights

    Toy Q and K from measured patch darkness and row position. Values are patch darkness; no learned weights or model explanation.

    Select a query patch and compare two computed attention heads. The outlined cell is the query; darker cells have greater weight, normalized relative to the largest weight in that map.

    For the ink head, q = [6dquery, 0, 0] and kj = [6dj, xj, yj], where d is measured darkness. Position coordinates do not influence that dot product. For the row head, q = [16yquery, −8, −8yquery2] and kj = [yj, yj2, 1]. Their dot product is −8(yquery − yj)2. Both heads use dk = 3. The weighted output shown is a single scalar darkness value.

    A trained head replaces those hand-picked rules with learned projections over much richer features. Different heads can form different mixtures, and later layers see features already informed by earlier mixtures. One head may connect characters along a line; another may gather broad document context. Such descriptions summarize observed patterns, not permanent job assignments. Attention heads are not independently named OCR modules.

    The weighted sum is also only part of a transformer block. Heads are combined and projected; residual connections preserve and add information; a per-position MLP transforms feature combinations; normalization stabilizes activations. After several blocks, a patch representation can depend on distant evidence without becoming a literal transcription of that evidence. The decimal point might influence a distributed set of coordinates rather than one readable “decimal” slot.

    Why a bright spot can mislead

    Darcet and colleagues' 2023 study, “Vision Transformers Need Registers,” identified high-norm tokens in relatively uninformative background regions. The studied models appeared to repurpose some patch positions for internal computation. Their proposed register tokens provide additional learned positions for that work, improving the smoothness of feature and attention maps in the evaluated settings.

    A register token has no corresponding camera pixel. Like a [CLS] token, it adds a sequence position, but its intended role is spare internal workspace rather than one designated classification summary. It must be accounted for when counting encoder positions. Whether any such positions reach the LLM depends on feature selection. This is another reason to distinguish an encoder's sequence length from its number of spatial patches.

    For our receipt, a large activation on blank paper would not prove that the blank paper explains $12.50. Even a clean-looking attention map only shows one set of routing weights at one layer and head. To test reliance on the total line, change the visible amount while holding other evidence steady and see whether the answer follows it. Interventions give stronger evidence than an attractive heatmap alone.

    Level 06/09Runtime

    Learn what pictures and words share

    The architecture now lets pixels influence one another, but nothing has taught it what a receipt means. One useful training signal comes from paired images and text. A photo beside a caption gives a weak statement about their shared content. It rarely labels every pixel or transcribes every line. Across many pairs, however, matching image and text representations becomes a useful problem to learn.

    CLIP trains an image encoder and a text encoder to make matching pairs similar in a shared embedding space. It uses normalized whole-image and whole-text vectors, so their dot product is cosine similarity. The representation supports comparing an image with candidate text descriptions. A contrastive image encoder can later supply spatial features to a generative VLM, but the contrastive objective itself does not generate an answer.

    Put N matched image-text pairs in a batch. Encode the images into unit vectors ui and the captions into unit vectors vj. Multiply every image against every caption, giving an N × N similarity matrix. The diagonal contains the provided matches. Off-diagonal entries are treated as negatives for this batch, even though some can be semantically compatible: two café photos may both fit “a receipt on a table.”

    A learned positive logit scale sharpens those similarities. Write the trainable scalar as a, and compute scale = exp(a) = 1/τ. The OpenAI implementation initializes a = log(1/0.07) ≈ 2.65926, giving temperature τ = 0.07 and scale ≈ 14.29. The CLIP paper's training recipe caps the effective scale at 100 to limit instability. That cap applies to exp(a), not to a; a scale cap of 100 corresponds to a ≤ log(100) ≈ 4.60517.

    For one image, a softmax over the row asks which caption belongs to it. For one caption, a softmax over the column asks which image belongs to it. The loss averages the negative log probability of the diagonal in both directions. The lab lets you set τ directly so you can see the effect of that rescaling on a fixed batch.

    sij = ui · vj / τ
    L = −(1 / 2N) Σi=1…N [log(exp(sii) / Σj exp(sij)) + log(exp(sii) / Σj exp(sji))]

    Unit vectors in two dimensions, chosen for this toy batch. Cells show cosine similarity and image-to-text probability; the loss uses both directions.

    A three-pair batch computes its cosine matrix and symmetric loss live. Swapping two captions keeps the diagonal training targets fixed, making those pair labels wrong.

    This is the symmetric contrastive cross-entropy often described as an InfoNCE-style objective. A low temperature exaggerates differences between scores. If the correct entry is already the largest, sharpening can lower the loss. If a wrong entry wins, sharpening can make the loss much worse. Temperature is not a universal confidence knob. It changes the penalty attached to the current geometry.

    As a sanity check, if all N scores in a row are equal, every caption has probability 1/N and the negative log probability is log N. If every row and column is uniform, the symmetric loss is also log N. For N = 3 that is about 1.09861 nats. Perfectly separated diagonal matches approach zero loss as their relative logits become large. Those checks follow directly from the equation and help catch normalization mistakes.

    Zero-shot classification is text retrieval

    Once both encoders share an embedding space, you can classify an image without fitting a new classifier on that class list. Write candidate descriptions such as “a photograph of a receipt,” encode them, and select the largest image-text similarity. The prompt wording supplies context for an otherwise ambiguous label. Averaging suitably constructed prompt embeddings is another option, but prompts can change the ranking.

    a photograph of a receipt0.999
    a photograph of a café0.878
    a photograph of a street0.170
    Zero-shot selection on the same two-dimensional toy geometry as the batch lab. The receipt image has angle zero; each candidate text vector has the angle used to compute its cosine score.

    The highest-scoring description says what kind of image this resembles. It does not provide the receipt's total. To return arbitrary text, a generative model needs a path from visual features to language predictions and training examples that reward using that path. A strong receipt classifier can still be a poor receipt reader. Whole-image semantic similarity can succeed while ignoring the exact final digit.

    SigLIP removes the shared denominator

    SigLIP's sigmoid loss scores each image-text pair as a separate binary match problem. Let zij be +1 for provided matches and −1 otherwise, and let ℓij = t(ui · vj) + b with learned scale t and bias b. Using the paper's normalization, L = −(1/N)Σij log σ(zijℓij). The sigmoid σ(a) = 1/(1 + exp(−a)) replaces the row and column softmax normalizers.

    The sigmoid model also learns its scale in log space: t = exp(t′). In the published SigLIP training recipe, t′ starts at log(10) ≈ 2.30259, so the effective scale t starts at 10; bias b starts at −10. Initializing t′ to 10 would instead produce exp(10) ≈ 22,026, a very different starting point.

    Why such a negative bias? An N-pair batch has N matching pairs and N(N − 1) negative pairs. At near-zero initial cosine similarities, b = 0 predicts a match probability near 1/2 for every pair. The many negative gradients can then overwhelm the positive signal. With b = −10, σ(b) ≈ 0.0000454: training starts with a low match prior and avoids that large early correction. The bias remains learnable; −10 is an initialization, not a fixed threshold or the exact prior for every batch size.

    Pairwise terms can be evaluated in chunks without keeping every similarity for a global softmax denominator. Distributed training still needs communication to expose pairs, but can circulate embedding chunks and accumulate loss locally. Evaluating all pairs remains quadratic in batch size; removing a denominator does not remove the pairs. Scale and bias also matter because a batch has many more negative pairs than positive ones.

    Calling this “learning without labels” needs one distinction: no fixed manually annotated category vocabulary is required, but the pairing is supervision. Bad captions and mismatched pairs corrupt it. For our receipt, a caption about a pleasant breakfast can teach scene semantics while supplying no reason to preserve $12.50. OCR and visual instruction data can teach the more exact task we actually need.

    Check yourselfYou lower the temperature when a wrong caption has the largest similarity. What can happen to the CLIP loss?

    Image-text contrastive learning aligns representations. Answering an arbitrary question additionally needs a generative connection and training that rewards the right visual evidence.

    Level 07/09Runtime

    Give the language model eyes

    The vision encoder has produced a sequence of useful features, but its coordinate system belongs to its own training. A decoder expects vectors with its own width and statistical structure. A The trainable component that converts visual encoder outputs into a representation a language model can consume. bridges the two. Its design sets both how features are transformed and how many visual positions the language system processes.

    The simplest connector is a linear layer applied independently to every spatial output. If the image features have width 1,024 and the decoder has width 4,096, its matrix has shape [1024, 4096]. For 576 patch outputs, multiplying [576, 1024] by that matrix produces [576, 4096]. The width changes; the sequence length stays 576. The matrix contains 4,194,304 weights, or 8 MiB if stored at two bytes per weight, plus any bias.

    The original LLaVA visual instruction-tuning work connects a vision encoder to a language model with a trainable projection. LLaVA-1.5 uses an MLP connector, adding a nonlinear transformation. The projector builder implements linear and multi-layer GELU variants. Adding layers alone still does not reduce the number of visual positions: token compression requires an operation across positions.

    Application message

    User: [receipt image]
    What is the total?

    Vision encoder output

    576 spatial vectors
    [576, 1024]

    Connector output

    [576, 1024] × [1024, 4096]
    = [576, 4096]

    Decoder input embeddings
    userv0 v1 … v575What is the total?assistant

    Then generate text: “The total is $12.50.”

    1. The image travels as image data beside your text. A chat template supplies role and boundary markers.
    2. Illustrative CLIP ViT-L/14@336 path: 24 × 24 spatial outputs, each 1,024 values wide; [CLS] is dropped.
    3. A linear connector changes the feature width to an illustrative 4,096-wide decoder. A two-layer MLP adds a nonlinearity.
    4. The placeholder expands into continuous visual embeddings. The shown answer is the target for our receipt, not browser inference.
    1 / 4
    Step through one illustrative 576-patch input as it becomes decoder embeddings. The dimensions follow a CLIP ViT-L/14@336 encoder and a 4,096-wide decoder with a linear connector.

    The application may place one image marker in a text template, but that marker can expand into hundreds of continuous embeddings. The decoder never runs byte-pair encoding on the RGB array. Text token embeddings and projected image features are assembled into a single input-embedding sequence, with role and modality boundaries where the model expects them. The LLaVA multimodal input preparation performs this replacement and excludes image positions from the text-target loss.

    Training the generative connection uses image-question-answer examples. For each answer token, minimize its negative log probability conditioned on the visual features, question and earlier answer tokens. Image features are context rather than text targets. Gradients can update the connector, decoder and encoder, depending on which parts are frozen. A frozen encoder still supplies features from which the connector can learn. This objective rewards an answer that uses the image only when the training examples make the image necessary; shortcuts in the data can still teach a model to guess.

    Those positions can carry visual information without having text IDs that mean what they contain. The output vocabulary still contains text tokens. At the first assistant position, the decoder computes logits for the next text token conditioned on the visual prefix and question; subsequent answer tokens condition on that evidence and the answer generated so far. Copying a number is learned behaviour, not a guaranteed memory read from a receipt field.

    Change the width, or change the length?

    BLIP-2's Q-Former replaces one-output-per-patch projection with learned queries. In the paper's ViT-L/14 example, a 224 × 224 image supplies 256 spatial patches plus [CLS]: visual features of shape [257, 1024]. The Q-Former begins with 32 learned query vectors, each 768 values wide. These queries interact through self-attention, then read image features through cross-attention inserted every other transformer block.

    Flamingo's Perceiver resampler produces 64 visual outputs. Its published models use a resampler width of 1,536, so the output is [64, 1536], regardless of the input feature-map length. Those latents feed gated cross-attention in the language model, rather than expanding a text marker into 64 ordinary decoder-prefix positions. Both systems bound the output length, but expose their visual summaries to language processing through different paths.

    A fixed latent count bounds the interface cost even when the visual input grows. It also creates a bottleneck: a longer receipt still has to pass through the same number of summaries. Cross-attention architectures keep those visual latents in a separate path, so their memory and compute accounting differs from inserting every image feature into the decoder's self-attention sequence. Always identify which architecture a quoted token count describes.

    Spatial merging takes another route. Gather adjacent 2 × 2 visual features and concatenate their channels. A [H, W, D] grid becomes [H/2, W/2, 4D] before projection. That space-to-depth rearrangement, also called pixel unshuffle, keeps the same number of scalar values while reducing the spatial position count fourfold. A later projection can compress the wider channels. Averaging four neighbours instead would discard their internal distinctions immediately.

    ConnectorWhat happens to lengthReceipt trade-off
    Linear / per-token MLPOne output per input positionPreserves spatial slots; decoder pays for each
    Learned query resamplingA chosen latent countBounded interface; long documents share a bottleneck
    2 × 2 spatial mergeA quarter as many positionsLocal grouping; channel projection must preserve fine detail

    Resolution is part of the model interface

    A square photo and a long receipt put different pressure on a fixed crop. Dynamic-resolution systems spend more positions where additional spatial samples can help. LLaVA-NeXT's January 2024 release uses multiple higher-resolution views. For a precise calculator, choose the grid candidates from a particular configuration rather than assuming every version has the same maximum size.

    The calculator's LLaVA-NeXT path uses the five pinpoints in llava-v1.6-mistral-7b-hf. These HF config pairs use (height, width), in their listed order. Its selection rule maximizes effective image area, capped at the original image's area, then minimizes wasted area; exact ties retain the first candidate. For a 336 × 336 input, the first pair (336, 672) therefore means a 672-wide, 336-high canvas. The chosen padded canvas becomes 336-pixel crops, each yielding 576 spatial features, alongside a global view. The lab shows both the raw crop-feature count and the packed decoder length, following Transformers v4.45.2 with default feature selection and newline embeddings enabled.

    Qwen2-VL's 2024 architecture instead handles a variable patch grid and merges neighbouring groups of four before the LLM. Its M-RoPE partitions rotary dimensions for temporal, vertical and horizontal coordinates; for a still image the temporal coordinate is constant across its patches. Distinguishing those axes helps encode layout without treating a row boundary as an ordinary single-axis step.

    Enter an input size and compare exact grid arithmetic for three named processing paths. LLaVA shows raw crop features and packed decoder positions; Qwen shows merged image features sent to the LLM.

    Pack the receipt: 2,880 features become 2,160 positions

    Our 640-wide, 960-high receipt selects a 672 × 672 canvas. Four local 336 × 336 crops plus one global view yield 5 × 576 = 2,880 raw features after dropping each crop's [CLS]. Projecting their channel width does not change that count. Reassemble the four local 24 × 24 grids into one 48-row, 48-column feature map; keep the global view's 576 vectors separate.

    The reference unpad_image and pack_image_features then remove padding from the local grid. Its height-based scale is 48/960 = 0.05, giving retained width floor(640 × 0.05) = 32. Removing eight feature columns from each side leaves 48 × 32 = 1,536 local vectors. Append one learned image_newline vector to each of the 48 rows: 48 × 33 = 1,584. Prepend the global 576 vectors to obtain 2,160 decoder positions. The newline vectors are visual sequence entries, not generated text line breaks.

    Unpadding uses symmetric integer padding, so a rounded target width can leave one extra column when the removed width is odd. The calculator follows that slicing rule instead of multiplying by an idealized aspect ratio.

    The Qwen path follows Transformers v4.45.2's Qwen2-VL smart_resize: round dimensions to multiples of 28, then adjust them to its stated pixel bounds. For a 640 × 960 receipt, it yields 644 × 952: a 46 × 68 encoder patch grid, then 23 × 34 = 782 merged visual positions. Special image-boundary tokens are additional. The static-image processor also accommodates the vision encoder's temporal patch dimension by repeating the frame; that does not double the resulting spatial token count.

    A connector decides how much visual detail the language model receives and how many positions it must pay for. Width conversion, spatial merging and query resampling solve different parts of that problem.

    Level 08/09System

    Keep enough evidence to answer

    The model now has a route from photo to answer. The serving system still has to choose how much of the photo reaches that route. For our receipt, preserving the whole page is necessary but insufficient: the final amount also has to be large enough to remain legible. A high-resolution upload can become a low-resolution model input, and a large token budget can still be spent on irrelevant table texture.

    A useful test is to inspect the processed image at its actual input resolution. At the sample's native size, the large $12.50 glyphs are ten pixels high. If you reduce the image width from 128 to 32 while preserving its aspect ratio, they are only 2.5 pixels high. The network receives sampled intensities, not the vector outlines from which the glyphs were drawn. At that scale, several distinct printed amounts can produce nearly the same small pattern.

    Built-in receipt

    Downsampled, enlarged

    Preparing the receipt…

    Browser downsampling with smoothing, nearest-neighbour enlargement. Read the pixels yourself; this panel runs no OCR model.

    Reduce the receipt and enlarge the result without smoothing. The number of retained samples changes; enlargement only makes those samples easier to inspect.

    The readout tracks the expected height of the total's glyphs before enlargement. At non-integer sizes, some strokes become partial intensities rather than clean black samples. This is why small-text recognition can fail abruptly around a resolution threshold: one extra sample can preserve a decimal point or distinguish the opening of a 2 from a 7.

    When the scene is clear and the answer is wrong

    Semantic recognition and exact reading make different demands. A heavily reduced receipt still looks like a receipt. An encoder trained on broad captions can preserve paper shape and café context while discarding character-level distinctions. Increasing resolution can supply missing samples, but the learned visual features and connector must also preserve them. A system tuned mainly for describing photographs may use fine print less reliably than one trained for documents.

    Counting is another trap. Attention aggregates features, and pooled summaries are well suited to presence: “there are cups here.” Exact counting needs individual instances, boundaries and a way to avoid counting the same item twice. On a receipt, asking how many purchased items appear requires distinguishing item lines from tax and subtotal lines. Recognizing all the words does not automatically establish the correct set of things to count.

    Hallucination appears when the answer supplies unsupported details. The language model has strong expectations about receipts: taxes, tips, familiar price formats and likely totals. When visual evidence is weak, those expectations can dominate. The POPE study of object hallucination evaluates related failures in which VLM answers assert objects unsupported by images. A fluent sentence is not a measurement of the strength of the visual evidence.

    Even a visible amount can be associated with the wrong label. Our subtotal of $10.50 is entirely real, clearly printed and financially plausible. Returning it for “total” is a binding error, not invented OCR. Separating these failure classes matters because their remedies differ: sharper input helps ambiguous glyphs, while examples that distinguish labels and layout help selecting the right field.

    Observed failureUseful interventionWhy it can help
    Total is absentPreserve the full page or add the missing cropRestores the required evidence
    Digits merge togetherSupply a closer crop with enough native detailSpends samples on characters
    Subtotal returnedKeep the amount and its label togetherPreserves the field relationship
    Confident invented amountRequire transcription plus location, allow “unreadable”Makes an unsupported answer easier to detect
    Arithmetic is wrongValidate extracted numbers with explicit arithmeticSeparates reading from calculation

    A practical receipt pipeline can keep a full-page overview and an additional close crop of the total region. The overview preserves document identity and surrounding structure; the crop supplies local samples. Keep labels with amounts so that an isolated $10.50 does not become ambiguous. Simply resizing a blurry original upward adds positions but no new optical detail.

    For repeated use, test the actual requested behaviour: exact amount and currency, correct distinction between total and subtotal, and a sensible response to unreadable images. Include crumpled pages, long receipts, faint print, rotated captures, unfamiliar tax layouts and amounts that differ by one digit. Change the visible total while keeping the rest of an example constant. An answer that follows the changed amount gives evidence of grounding.

    When exactness matters, compare extracted fields with a second reading path or a deterministic check. Here 4.50 + 6.00 = 10.50 and 10.50 + 2.00 = 12.50. That consistency check can reject some mistakes, though a wrong transcription can still happen to balance. An evidence crop and the original image make failures inspectable. A format-valid JSON response alone only proves that a serializer accepted the answer.

    Check yourselfA model says $10.50, the clearly printed subtotal, instead of $12.50. What does that observation establish?

    Level 09/09Machine

    Pay for every visual position

    The resolution choices now have a physical bill. The vision encoder must compute image features, the connector must transform them, and the decoder must process the resulting positions before it can emit the first answer token. For our short question, image processing can create far more positions than text tokenization. The cost begins with the same multiply-and-add that appeared in the 3 × 3 filter.

    Patch embedding is a matrix multiply

    Imagine collecting all flattened patches as rows of X. With N patches and an RGB patch width p, X has shape [N, 3p2]. Multiplying by E of shape [3p2, D] costs approximately 2N(3p2)D floating-point operations, counting a multiplication and an addition separately. A kernel can perform the equivalent strided convolution without materializing that whole patch matrix.

    For a 224 × 224 image, p = 16 and D = 768, this is 2 × 196 × 768 × 768 = 231,211,008 FLOPs, about 0.231 GFLOP. Bias adds are small beside that product. The figure is an arithmetic count, not a runtime prediction: memory layout, precision, kernel selection and available parallelism determine how efficiently the hardware executes it.

    Transformer blocks add projections, attention and MLPs. In a conventional block of width D with an MLP expanded to 4D, the Q, K, V and output projections cost about 8ND2 FLOPs. The two MLP matrices cost about 16ND2. Full attention's score and value products add about 4N2D. These expressions omit normalization, activations, softmax and biases to expose the leading matrix work.

    ViT block matrix work ≈ 24ND2 + 4N2D
    Double both image sides → 4N → 4× linear work, 16× attention-pair work

    At modest N and large D, the projections and MLP can dominate FLOPs. “Attention is quadratic” does not imply that attention is always the most expensive operation. It tells you how one term scales. Increasing resolution eventually makes that term more significant, while every extra patch already adds projection and MLP work.

    The decoder pays again

    In an embedding-insertion VLM, image positions participate in the decoder's The initial forward pass over all supplied prompt positions, before autoregressive generation begins.. Let T be the text length and I the visual length. With ordinary causal self-attention, there are (T + I)(T + I + 1)/2 allowed query-key pairs per head. Relative to text alone, added pairs are that number minus T(T + 1)/2. Each position also passes through the decoder's projections and MLPs.

    The calculator counts the additional score and weighted-value matrix work across an illustrative 32-layer decoder. A causal attention pair contributes about 4D FLOPs for those two products across all query heads. A dense masked implementation may execute more work; kernels that skip causal blocks can avoid much of it. The reported total excludes the vision encoder, connector, projection and MLP work.

    KV heads (32 query heads)
    Visual KV cache
    Added causal attention
    Prefill length

    Illustrative decoder: batch 1, 32 layers, width 4,096, head dimension 128, BF16 KV, 64 text positions. FLOPs count QK and AV over permitted causal pairs, one multiply-add = 2 FLOPs.

    Change image positions and KV-head count to see decoder memory and causal attention work. Grouping KV heads reduces cached bytes while preserving the query-head attention-pair count.

    The decoder usually retains keys and values for every prefix position during generation. Its visual share of the KV cache occupies 2 × I × L × HKV × dhead × b bytes: two tensors, I positions, L layers, HKV key/value heads, dhead coordinates and b bytes per stored number. At I = 576, L = 32, HKV = 8, dhead = 128 and BF16 b = 2, that is 75,497,472 bytes, exactly 72 MiB for one request.

    Those cache entries let the decoder reuse previous computations while generating “The total is $12.50.” The image need not run through the encoder again for every answer token. Each new token still attends to available visual prefix keys and combines their values. Grouped-query attention shares KV heads among more query heads; the calculator's full-head alternative uses four times as many cached bytes. Serving systems must budget that storage across simultaneous requests.

    Move fewer bytes between multiplies

    On a GPU, matrix multiplications are tiled. Blocks of activations and weights move from high-bandwidth device memory into smaller on-chip storage, where many products reuse them. Matrix units accumulate partial results before the kernel writes them out. Large, well-shaped operations can reuse each fetched weight across many patch or prompt rows. Many tiny crops can add scheduling overhead even when their arithmetic counts look manageable.

    The same patch matrix is consumed in tiles. Reusing an activation or weight on chip reduces memory traffic without changing the matrix product.

    FlashAttention applies this principle to attention: compute score blocks and normalize incrementally rather than writing the entire attention matrix to device memory. It preserves exact attention up to numerical rounding while reducing memory traffic and intermediate storage. It does not remove the pairwise arithmetic growth or shrink the persistent KV cache.

    This gives the receipt's final route. Encoded file bytes became RGB samples. A processor preserved some samples and discarded others. Patch projections produced spatial features; attention let them interact; training made those features useful; a connector placed visual information where the decoder could use it. Cached keys and values kept that evidence available while the answer emerged one text token at a time.

    The model's ability to report $12.50 rests on every layer of that route. Better training helps it use evidence. Better resolution policy helps it retain evidence. Better kernels help it afford the evidence. At the bottom, the last dollar sign is supported by bytes arriving at multiply units soon enough, and by the right distinctions surviving all the way from the photo.

    High resolution costs three times: more work in the image encoder, more decoder prefill work if visual length grows, and more persistent cache for visual prefix positions.

    End of Vision models · 9 levels, Idea to Machine

    Where to next

    Browse all 34 divesAll topics