AI modelsDive 05 of 06
How a speech model hears
Say “We see a cat.” Follow the air pressure through samples, Fourier transforms, mel filters and a Whisper decoder, all the way to the bytes and arithmetic that turn sound into text.
- Levels
- 9
- Reading
- 38 min
- Depth
- Idea to Machine
- Published
One utterance · three representations
“We see a cat.”
Formant sketch with vowels and noise bursts; the words are its synthesis score. No recognizer runs on this page.
Computing the sentence in small batches…
A sentence with no spaces
Say We see a cat.
A microphone does not receive four words. It receives a continuously changing pressure, including your breath, the room’s echo, a fan, and perhaps another person. There are no spaces in that pressure. There is no punctuation either. A speech recognizer has to recover a sequence of written symbols from a sequence of physical measurements, even though the two sequences have different lengths and no obvious shared clock.
The output you want is deceptively small: We see a cat.
Perhaps you also want a language label, a start time, an end time, and word timings for captions. The hard part is deciding which details of the sound explain the words and which details to ignore. The same sentence changes when a child says it, when you whisper it, when you speak into a laptop across the room, or when you run its recording through a telephone connection.
Automatic speech recognition: converting recorded or arriving speech into a written transcript., or ASR, learns that many different sounds can have the same transcript. It also learns that similar sounds can mean different things. We see a cat
and We see a cap
differ mostly near the end. Losing a short consonant burst may change the noun. Hearing more of the sentence can resolve an ambiguity, but waiting for more sound adds latency.
A useful model of the problem
Write the recording as x and a possible transcript as y. The recognizer tries to choose a likely y given x. In a sequence-to-sequence model, it factorizes that probability into next-token predictions. The next written piece depends on the audio and on the pieces already chosen. This combines two sources of evidence: what the sound supports and what makes sense as language. A fluent sentence can be acoustically wrong.
Whisper is a concrete example of this design. Its encoder reads a representation of the recording; its decoder writes text while attending to that encoder’s output. The original Whisper paper describes training on 680,000 hours of multilingual and multitask supervision. The important training signal is paired audio and text. The model learns representations and spelling jointly rather than requiring a hand-written pronunciation rule for every possible word.
Our running sentence is generated locally from a tiny synthesis score. Harmonics sketch the vowels; short noise intervals sketch /s/, /k/, and /t/. It sounds mechanical because it leaves out most of the articulatory detail of a human voice. That simplicity makes the evidence easy to locate. The waveform and mel values on this page are computed from those samples. The decoder probabilities later are a separate, explicitly labelled teaching model.
Those tidy gaps belong to our score, not to speech in general. In ordinary conversation, adjacent sounds overlap as the mouth anticipates the next one. This is coarticulation: the vowel before /t/ already carries clues about the coming consonant. A recognizer benefits from context because a useful acoustic cue can occur before the spelling unit you associate with it.
A transcript is also a convention. Should uh
appear? Should a number be written as four
or 4
? Should a laugh receive a tag? Training data teaches those choices. The microphone alone cannot tell you whether to capitalize We
, so written form is partly an inference about the intended output format. Evaluating ASR therefore requires a normalization policy as well as a reference transcript.
Speech recognition recovers a short symbolic sequence from a long, continuous, ambiguous signal. Sound supplies evidence; learned language helps interpret it.
Catch the moving air
Before a model can interpret the recording, the moving air must become numbers. Sound is a variation in pressure around the ambient pressure. A microphone’s diaphragm responds to that variation, and its transducer produces an electrical signal. An analog front end amplifies and filters it. An An analog-to-digital converter measures a continuous voltage at discrete times and maps each measurement to a numeric code. then measures that voltage on a clock.
Two choices are independent. Sampling rate tells you how often to measure: 16,000 samples per second means one measurement every 62.5 microseconds. Bit depth tells you how finely to represent each amplitude. A 16-bit code has 65,536 possible values. Increasing the sampling rate gives you finer time spacing; increasing the bit depth gives you finer amplitude spacing. Neither repairs a clipped microphone signal.
What sampling preserves
A sine wave has a frequency, an amplitude, and a phase. Sampling writes down its value at equally spaced times. For a signal band-limited below half the sample rate, ideal reconstruction can recover the continuous signal from those samples. That half-rate boundary is the Nyquist frequency. At 16 kHz, it is 8 kHz. The strict band-limit matters: a tone exactly at the boundary can disappear for some phases.
Above that boundary, different frequencies produce the same sample sequence. A 6 kHz sine sampled at 8 kHz folds to 2 kHz, with a phase inversion in this example. This is aliasing. Once two input signals have produced identical samples, no downstream neural network can generally determine which was present. The analog anti-alias filter removes the out-of-band energy before the ADC samples it.
falias = |((f + fs/2) mod fs) − fs/2|
- Original tone
- Aliased reconstruction
- ADC samples
Five byte pairs encode −1, −0.5, 0, 0.5 and the positive limit in PCM16. The bit-depth control changes the plotted ADC levels.
Try 6,000 Hz at 8,000 samples/s, then switch to 16,000. The apparent 2 kHz tone becomes a representable 6 kHz tone. At 8,000 Hz input and 8,000 samples/s, this zero-phase sine produces all zeros. The sampling clock repeatedly catches the same point in its cycle. A higher sample rate prevents this particular collision, but real resampling still needs a low-pass filter before reducing the rate.
From a code to bytes
Signed PCM16 conventionally stores amplitudes as integers from −32,768 to 32,767. In little-endian order, −32,768 is 00 80, −16,384 is 00 C0, zero is 00 00, and 16,384 is 00 40. The positive maximum is FF 7F. Dividing a decoded integer by 32,768 gives a floating-point value in [−1, 1), which is the scaling used by Whisper’s file loader.
Mono PCM16 at 16 kHz needs 32,000 bytes per second. A 30-second window is 480,000 samples and 960,000 bytes before any container header. Stereo would double that storage without necessarily doubling the speech information. Whisper’s loader asks FFmpeg to decode, downmix to one channel, and resample to 16 kHz. An already-decoded array must arrive with the expected rate; an array of numbers does not carry a sampling clock by itself. See the audio loader.
Sixteen kilohertz preserves a useful speech band while keeping the input manageable. Higher-frequency energy can help distinguish consonants or improve fidelity, so 16 kHz is an engineering choice rather than a claim that human speech ends at 8 kHz. Whisper was trained with that representation. Feeding 48 kHz samples into the same code without resampling would make a second of speech occupy three seconds on the model’s assumed clock.
Quantization introduces rounding error. For a full-scale sine under the usual ideal quantization assumptions, the signal-to-quantization-noise ratio is approximately 6.02b + 1.76 dB for b bits. Real microphones have analog noise, rooms have background noise, and speech rarely stays at full scale. A gain setting that avoids clipping and leaves usable headroom often matters more than adding nominal ADC bits beyond the recording’s effective noise floor.
Listen to one instant
The sample sequence preserves timing, but nearby numbers do not directly describe a vowel. A vowel contains many harmonics shaped by resonances in the vocal tract. We want to expose those frequencies while retaining when they occurred. The standard compromise is to analyze a short, overlapping frame at a time: long enough to contain several pitch cycles, short enough that its spectral character is approximately stable.
Whisper uses a 400-sample window at 16 kHz: 25 milliseconds. The hop is 160 samples: 10 milliseconds. Adjacent windows overlap by 240 samples, or 60%. A frame is a view onto a short region of the same continuous signal, not an isolated syllable. Our 125 Hz synthesized pitch has a period of 8 ms, so one frame contains a little over three pitch periods.
The Fourier transform asks a precise question
For each candidate frequency, multiply the frame by a cosine and a sine and add the results. A strong match survives the summation; a mismatch tends to cancel. The two sums preserve phase as a complex coefficient. For N samples, the discrete Fourier transform is:
X[k] = Σn=0N−1 x[n]w[n] exp(−i2πkn/N)
Here w[n] is a window function. The index k corresponds to frequency kfs/N. With N = 400 and fs = 16,000, bins are 40 Hz apart. A real-valued input has conjugate symmetry, so only bins 0 through 200 are needed: 201 complex values from zero through 8 kHz. Whisper then takes squared magnitude, |X[k]|2, to obtain a power-like spectrum and discards explicit phase.
The direct DFT evaluates every sample against every frequency, requiring work proportional to N2. A fast Fourier transform reorganizes the same sums. A radix-2 FFT repeatedly splits even- and odd-indexed inputs, reuses smaller transforms, and combines them with complex rotations called twiddle factors. That reduces work to order N log N when N is a power of two.
Four hundred is not a power of two. The implementation here uses Bluestein’s identity to turn the exact 400-point transform into a convolution, then evaluates that convolution using 1,024-point radix-2 FFTs. It agrees with a direct DFT in the numerical tests. Simply padding the frame to 512 and treating those bins as Whisper’s bins would change the frequency grid and the mel filterbank multiplication.
A 25 ms listening window
Computing the sentence in small batches…
Why the window changes the picture
A DFT effectively analyzes a periodic repetition of its finite input. Cutting a waveform at arbitrary points can put a discontinuity where the end joins the beginning. A rectangular window keeps the frame unchanged, including that discontinuity. Its energy spreads into distant frequency bins: spectral leakage. A tone between FFT bins leaks even when the source is perfectly periodic over a longer interval.
The Hann window tapers both ends toward zero. Whisper uses the periodic convention w[n] = 0.5 − 0.5 cos(2πn/N), produced by torch.hann_window(400). It reduces distant sidelobes at the cost of widening the main lobe. Switching windows changes both the leakage pattern and the gain. The plot normalizes each spectrum to its own peak so the leakage difference is easier to compare.
Choose the 437 Hz tone. Its frequency is near the 440 Hz bin, but it is not exactly on that bin. With a rectangle, the small mismatch creates a long tail. With Hann, the energy spreads across a wider local peak and falls off much faster farther away. A narrow peak is therefore not automatically a better estimate: resolution and leakage are different properties.
Scrub back to /s/ around 0.72 s. The broadband noise differs sharply from the harmonic vowel around 0.92 s. A single frame can reveal that distinction. It cannot identify the whole sentence, and increasing frequency resolution by taking a much longer window would blur the short transitions we need for consonants. The frame/hop choice sets a time-frequency compromise before the learned model begins.
The FFT does not recognize words. It exposes the frequencies inside a short window, while overlapping windows preserve how those frequencies change over time.
Build the picture of sound
Stack the 201-bin spectra in time and you have a spectrogram. Whisper uses a smaller, perceptually motivated frequency representation. Closely spaced low frequencies receive separate channels; broader high-frequency intervals share channels. A A weighted sum of power-spectrum bins arranged on a perceptual frequency scale; overlapping triangular weights define the bands. performs this reduction with a fixed matrix, before any learned weight is applied.
The name mel does not uniquely specify the matrix. One common formula, used by HTK, is m = 2595 log10(1 + f/700). Whisper’s stored matrices instead come from librosa.filters.mel with its defaultSlaney scale and area normalization. Substituting the HTK formula produces a plausible-looking spectrogram with different numbers. The model expects the numbers it was trained on.
Slaney’s scale is linear below 1 kHz: m = f/(200/3). Above 1 kHz, it is logarithmic: m = 15 + ln(f/1000)/(ln(6.4)/27). The units differ from HTK’s; only the resulting frequency placement matters here. We take equally spaced mel coordinates between zero and 8 kHz and convert them back to hertz to place each filter’s left edge, center, and right edge.
From triangles to energy
Each filter rises linearly from zero at its left edge to one at its center, then falls to zero at its right edge. Its samples are evaluated at the FFT frequencies, 0, 40, 80 Hz and so on. Slaney normalization multiplies the triangle by 2 divided by its full width in hertz. This gives equal continuous area to narrow and broad filters; it does not force their sampled coefficients to sum to one.
For mel band m and time t, compute Mm,t = Σk Fm,k|Xt[k]|2. The filter matrix has shape 80 × 201 for the earlier models and 128 × 201 for large-v3 and turbo. Neighboring filters overlap, so a frequency component can contribute to more than one channel. More mel channels do not create new raw frequency information; they redistribute the fixed FFT-bin evidence.
240 frames · one synthetic utterance
- Frame power spectrum
- Inactive filters
- Selected filter
Filter curves use relative height here; the calculation includes Slaney area normalization. Brightness is normalized log power.
Computing the sentence in small batches…
Whisper’s exact logarithm and normalization
Power varies across many orders of magnitude. A logarithm makes weak features visible beside strong ones and makes multiplicative amplitude changes look like additive offsets. Whisper first clamps each mel energy to at least 10−10, takes log10, and then floors all values at the maximum log energy minus eight. Finally it adds four and divides by four. These constants are visible directly in audio.py.
L = log10(max(M, 10−10))
L′ = max(L, max(L) − 8)
Z = (L′ + 4)/4
| Mel power | After log clamp | After max−8 floor | Final value |
|---|---|---|---|
| 10−12 | −10 | −6 | −0.5 |
| 1 | 0 | 0 | 1 |
| 100 | 2 | 2 | 1.5 |
This is not a min-max mapping into [0, 1]. The result can be negative or greater than one, as the table shows. The dynamic-range floor covers eight base-ten orders of power, equivalent to 80 dB. Because the reference is the maximum of the supplied spectrogram tensor, changing the analyzed audio can change the floor elsewhere. Normalizing isolated chunks independently can therefore produce different low-energy values than slicing a spectrogram computed for the complete recording.
The reference STFT centers its windows, using reflection padding at the input edges, and drops the final time column. For exactly 480,000 waveform samples, the retained result has 3,000 frames. The local 2.4-second example uses the same centered framing, periodic Hann window, exact 400-point spectrum, and log transform, producing 240 frames. Only the short example is rendered; a real fixed-context encoder receives the padded 30-second input.
There are no learned features yet. The triangles, logarithm, hop, and normalization are fixed preprocessing. The spectrogram emphasizes the changes the network needs to interpret: stable resonant bands during vowels, energy high in the spectrum during noisy consonants, and sudden transitions around releases. Phase and some frequency detail have been discarded, so this representation is useful for recognizing speech without being an exact reversible encoding of the waveform.
Give every moment context
We now have a grid of real numbers indexed by time and mel channel. A speech encoder turns that grid into a sequence of learned feature vectors. Early features can combine neighboring frequency bands and nearby moments; deeper features can use the entire recording to interpret a local sound. The output still has a time axis, but its coordinates no longer have simple labels such as energy near 700 Hz
.
Whisper starts with two one-dimensional convolutions across time. The first maps the mel channels to model width d, with a three-frame kernel, stride one, and padding one. GELU supplies a nonlinear activation. The second again has a three-frame kernel and padding one, but uses stride two, halving the temporal length. Another GELU follows. For a 30-second input, 3,000 mel frames become 1,500 encoder positions. Width d depends on the model size.
80 × 300030 s at 100 frames/s; 128 channels for large-v3
d × 3000Kernel 3, stride 1, padding 1
d × 1500Kernel 3, stride 2, padding 1
1500 × dFixed sinusoids; unmasked attention over the whole window
1500 × dOne contextual vector per 20 ms position, after layer normalization
A convolution reuses the same small set of learned weights at every time offset. Each output channel is a weighted sum across the input channels and three adjacent frames. With two such kernels, the second layer’s local receptive field covers five input mel frames before attention begins. Those frames’ centers span 40 ms; including each 25 ms analysis window gives a nominal 65 ms span of waveform support away from boundaries.
Where the order comes from
Attention needs a way to distinguish earlier from later positions. Whisper adds fixed sinusoidal position vectors to the encoder features. Different coordinates oscillate at different rates, giving each time index a recognizable pattern. The decoder uses a different choice: learned positional embeddings for its text positions. Position schemes belong to the architecture; they are not interchangeable at inference time.
Each encoder transformer block contains self-attention and a feed-forward network, with layer normalization and residual additions. In attention, every position creates a query, key, and value. Dot products between its query and all keys determine a distribution over the other positions. The weighted sum of their values lets a local vector collect evidence from elsewhere in the recording. Multiple heads compute several such mixtures.
Encoder self-attention is unmasked. A position near the start can consult a consonant near the end. For our sentence, the representation around cat
can use the earlier a
, the final release, and neighboring silence. A feed-forward network then transforms each position separately, with a hidden width four times d. Residual paths preserve and refine the running representation through many layers. See AudioEncoder and ResidualAttentionBlock.
The vectors are trained to help the decoder predict the reference transcript. There is no requirement that one coordinate become a phoneme detector, and no fixed stage at which the network must explicitly spell /k/, /æ/, /t/. Useful phonetic, speaker, channel and contextual information can coexist in distributed features. Interpreting attention as a literal explanation of a word choice is therefore risky: a weighted mixture is a computation, not an annotated reasoning trace.
From mel frames to contextual vectors
| Model | Input | Conv k3, stride 1 | Conv k3, stride 2 → transpose |
|---|---|---|---|
| tiny | 80 × 3000 | 384 × 3000 | 1500 × 384 |
| base | 80 × 3000 | 512 × 3000 | 1500 × 512 |
| small | 80 × 3000 | 768 × 3000 | 1500 × 768 |
| medium | 80 × 3000 | 1024 × 3000 | 1500 × 1024 |
| large-v3 | 128 × 3000 | 1280 × 3000 | 1500 × 1280 |
| large-v3-turbo | 128 × 3000 | 1280 × 3000 | 1500 × 1280 |
Both convolutions use padding 1 and GELU. Small has 12 attention heads of 64 coordinates each; 12 × 64 = 768.
| Frame | Key = value | q · k / √2 | Softmax weight |
|---|
q: 1 × 2; K: 3 × 2; V: 3 × 2
softmax(qKT/√2): 1 × 3
Weighted output: 1 × 2
Computing three mel-frame vectors…
Two actual normalized mel channels, 9 and 25, stand in for projected features. Q, K and V use the identity here; Whisper learns projections and uses all 1500 positions.
For small, the convolution weights have shapes 768 × 80 × 3 and 768 × 768 × 3. At one time position, the first convolution combines 80 channels from each of three adjacent mel frames: 240 inputs per output channel. The second combines three width-768 vectors and keeps every other output. Transposing its 768 × 1500 result gives 1500 rows of 768 features for the transformer.
A real small-model head projects those features into 64-dimensional queries, keys and values. The toy above keeps only two mel coordinates and uses identity projections so every number remains visible. For the selected query, three scaled dot products become three softmax weights; those weights multiply the three value vectors to produce one output vector. Changing the query changes the mixture, while every row still sums to one. In the full encoder this operation runs for every query, head and layer, with 1500 possible keys instead of three.
Why a 30-second box?
The original model was trained on 30-second segments. That choice fixes a tractable attention length and a consistent training format while retaining enough context for phrases and short exchanges. The reference encoder checks that the post-convolution shape matches its 1,500-position embedding buffer. A 2.4-second waveform must therefore be padded for this path; a longer recording must be divided into windows.
Padding is not an instruction to spend 30 seconds waiting. A completed short clip can be padded immediately and processed faster than real time. It does mean the reference dense encoder still performs work for the fixed tensor shape. Likewise, padding waveform samples with zero before the STFT and padding already-normalized feature frames are distinct operations. The loader, feature extractor and transcription loop determine the convention together.
The time grid after downsampling is 20 ms, but an encoder vector’s information is not restricted to that instant. The convolution already combines neighboring frames, and full self-attention can mix the entire window. The grid locates positions; it does not promise an independent acoustic measurement or exact word boundary every 20 ms. That distinction matters when the decoder later produces timestamp tokens.
The encoder turns local spectral evidence into contextual vectors. It keeps 1,500 time positions per 30-second window, while letting each position use the entire window.
Write what was heard
The encoder has prepared evidence. The decoder now has to write. At each step it reads the tokens already supplied, computes a score for every possible next token, and converts those scores into probabilities. Its causal self-attention sees previous text positions; its cross-attention reads the encoder’s audio vectors. That second attention mechanism is how sound influences the otherwise familiar language-model loop.
For a transcript y with U tokens, the model assigns P(y|x) = Πu=1U P(yu|y<u, Enc(x)). Each factor depends on the audio and the chosen prefix. During training, the prefix comes from the reference transcript, and the loss penalizes low probability on the correct next token. During inference, the prefix contains the model’s own choices, so an early mistake changes the conditions for later predictions.
The prompt specifies what to write
Whisper’s multilingual format begins with <|startoftranscript|>, followed by a language token such as <|en|> and a task token. <|transcribe|> asks for text in the spoken language;<|translate|> asks for translation into English. Adding<|notimestamps|> requests text without the timestamp-token sequence. These are reserved vocabulary entries with learned behavior, not literal English instructions.
Ordinary text uses a byte-level BPE vocabulary. Common sequences can occupy one token; uncommon words can split into multiple pieces, sometimes across Unicode character boundaries. A leading space may belong to the next piece. Consequently four written words need not imply four decode steps. Punctuation and the end marker also require predictions. Multilingual versions through large-v2 use 51,865 vocabulary entries; large-v3 and turbo use 51,866. The exact IDs depend on the tokenizer version and language set. See tokenizer.py.
Prompt
<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>
Output prefix
We
Each appended token becomes an input for the next prediction.
The end token terminates this utterance.
- The first distribution chooses “We”. The acoustic encoder has already run.
- Two plausible verbs compete: “see” has probability 0.55, “saw” has 0.45.
- The next token is “ a”. Leading spaces are part of these toy text pieces.
- The acoustic evidence favours “ cat”.
- The decoder adds punctuation absent from the waveform.
- Greedy completes “We see a cat.” A width-two beam prefers “We saw a cat.” in this deliberately misleading toy.
Hand-authored probabilities and word-sized tokens, not a Whisper run. The two verbs expose the difference between a local choice and a sequence score.
Greedy decoding always appends the most probable next token. It is cheap, but it optimizes one local decision at a time. In the toy, see
wins 0.55 to 0.45, then its full-stop continuation receives only 0.6. That path has probability 0.33. The saw
path receives a full-stop probability of 0.99 and reaches 0.4455. Keeping both verb prefixes long enough exposes the better complete path.
Beam search retains several high-scoring prefixes, expands each, and prunes again. It sums log probabilities rather than multiplying tiny numbers, and implementations may apply length normalization when ranking completed sequences. It is still an approximation: a prefix discarded from the beam cannot recover later. A larger beam also costs memory and decoder work, and the highest model-probability transcript need not be the most faithful transcript.
Timestamps are a second output alphabet
With timestamps enabled, the decoder can emit tokens representing times from 0.00 to 30.00 seconds in 0.02-second increments. Text and timing markers share the autoregressive stream. Timestamp pairing and monotonicity rules restrict which tokens are legal during decoding. A possible teaching sequence is a start timestamp, the text We see a cat.
, and an end timestamp. Quantizing our score’s 2.13-second speech ending to the nearest grid point would give 2.14 seconds; that is an illustration of the grid, not a measured model prediction.
Word timestamps are a separate refinement in the reference implementation. They use selected cross-attention patterns and dynamic time warping to align text with audio. A 20 ms timestamp alphabet does not guarantee word boundaries accurate to 20 ms. Ambiguous releases, overlap, background noise and alignment heuristics can all shift a boundary. The time units are a representation choice, while alignment accuracy is an empirical property.
When the first decode looks wrong
The reference transcription loop can retry a window at temperatures 0.0, 0.2, 0.4, 0.6, 0.8 and 1.0. A positive temperature samples instead of taking a deterministic argmax; higher temperatures flatten the distribution. The loop retries when the output is unusually repetitive or its average token log probability is too low. For nonzero temperatures it removes beam search options; this is a change of search strategy, not a second pass through a better encoder.
| Signal | Default | Action |
|---|---|---|
| Compression ratio | > 2.4 | Retry at the next temperature; compressed repetition is suspicious |
| Average log probability | < −1.0 | Retry; the chosen sequence received weak model support |
| No-speech probability | > 0.6 | Skip unless average log probability is > −1.0 |
| Temperature schedule | 0.0 … 1.0 by 0.2 | Return the final attempt if no earlier attempt passes |
The compression utility actually uses zlib compression of UTF-8 text. The ratio is uncompressed byte length divided by compressed byte length. Repeated phrases compress well, so a high ratio detects some repetition loops. The no-speech branch can suppress fallback when both no-speech probability is high and average log probability is low. During the skip decision, a sufficiently high average log probability overrides the no-speech score. These interacting conditions are in transcribe.py, rather than inside the transformer layers.
Every retry is still a hypothesis about the same sound. Fluency, repetition checks and model confidence are useful signals, but they cannot prove that someone said the words. This becomes especially visible on silence, music, or unfamiliar acoustic conditions, where the decoder’s learned language patterns can overpower weak audio evidence.
Learn without a stopwatch
Whisper learns to emit text while attending to audio. A different family of recognizers starts from a more constrained question: which symbol, if any, should this acoustic position contribute? This leads to connectionist temporal classification, or CTC. It solves the training problem where you know the transcript but do not know the frame at which each letter or subword occurred.
Suppose a small model emits one probability distribution per frame over c, a, t and a blank symbol _. A path through those frame distributions is an alignment. The path _ c c a _ t should represent cat
, even though it has six symbols. CTC defines a collapse rule: merge adjacent repeated symbols first, then remove blanks.
The order matters. c c collapses to one c, whereas c _ c collapses to two. Removing the blank first would lose that distinction. Blank means emit no label at this frame
; it is not a space character and need not indicate acoustic silence. A voiced vowel can occupy many frames while only one of them emits its label.
Training sums over alignments
CTC assigns a path the product of its per-frame symbol probabilities. It assigns the transcript the sum of the probabilities of all paths that collapse to that transcript. Taking the negative natural logarithm gives the training loss. Picking only the best path would throw away other valid explanations of the timing and would produce a different objective.
P(y|x) = Σπ : B(π)=y Πt=1T pt(πt)
ℒCTC = −ln P(y|x)
Edit one alignment · _ is blank
Six toy frames stand in for the encoder sequence. Each row sums to one; the forward table sums all valid alignments for the selected target.
Enumerating every path grows exponentially with the number of frames. Dynamic programming avoids that explosion. Insert blanks around and between target labels, giving _ c _ a _ t _ for cat
. A forward value αt,s stores the probability mass that has reached state s at frame t while following a valid partial alignment.
To update a state, add incoming mass from staying at that state, advancing by one state, and sometimes skipping two states. The two-state skip is allowed only into a nonblank label different from the label two states earlier. Then multiply by the current frame’s probability for that state’s symbol. This restriction forces repeated target letters to have a separating blank instead of merging into one letter.
At the first frame, only the initial blank and first label can have probability mass. At the final frame, the path may end on the last label or its following blank, so sum those two states. The tiny lattice above uses ordinary double-precision probabilities. Production CTC typically works in log space with log-sum-exp to keep long products from underflowing. The recurrence is the same; the numeric representation changes.
Try the target ccat
. It needs two separate c emissions and therefore a blank between them. Edit the path to c _ c a t _ and it contributes to that target. The original _ c c a _ t does not. Raising blank probability helps some timing paths while taking probability away from every nonblank symbol, so transcript probability need not rise monotonically.
What came before, and what streams naturally
Older hybrid ASR systems often combined frame-level acoustic scores, a hidden Markov model for state transitions, a pronunciation lexicon, and a separate language model. That division made linguistic constraints explicit but required several separately engineered components. End-to-end objectives reduced the need for hand-aligned acoustic targets and allowed more of the pipeline to be learned jointly.
CTC assumes output labels are conditionally independent across frames once the encoder representation is given. The encoder itself may still use broad audio context. Greedily collapsing the most likely frame labels is fast, but the most likely path does not always yield the most likely transcript because several paths can support the same text. Prefix beam search and external language models can improve decoding.
RNN-T, the recurrent neural network transducer, adds an output-history predictor. An acoustic encoder tracks input time, a prediction network tracks emitted labels, and a joiner combines both to score a label or blank. Blank advances acoustic time; a label advances the output sequence without necessarily advancing input time. Training sums paths through this two-dimensional lattice. Modern transducers may use transformer-based encoders even though the historical name includes RNN.
| Family | Alignment mechanism | Output history | Streaming implications |
|---|---|---|---|
| Hybrid HMM | Paths through acoustic/lexical states | External language model | Incremental search is natural; pipeline is explicit |
| CTC | Blank, repeat collapse, summed frame paths | Optional external language model | Cheap incremental emission with a streaming encoder |
| RNN-T | Time × label lattice | Learned prediction network | Designed for incremental transduction |
| Whisper | Decoder cross-attention and timestamps | Autoregressive text decoder | Full-window encoder; wrappers must manage chunks and revisions |
The distinction is architectural, not a ranking of quality. A recognizer trained for live captions makes different tradeoffs from one transcribing a complete interview. The original CTC paper and Graves’s transducer paper define the training objectives; latency also depends on the encoder and serving policy wrapped around them.
Speak before the audio ends
A file already contains its future. A live microphone does not. That difference changes the system around the model. On a recording, you can inspect the next 30 seconds and use later words to disambiguate earlier ones. In a call, a caption for We see
might need to appear before the speaker has finished a cat
. The system must choose when to publish a hypothesis and when it is stable enough to call final.
The reference Whisper transcription loop processes long audio by taking windows of up to 30 seconds and using decoded timestamps to advance. Previous text can be provided as a prompt for the next window to improve continuity. This is file transcription, not a permanently stateful causal audio encoder. Reusing a text prompt does not let the original unmasked encoder accept arbitrary future samples without reconsidering its earlier representations.
Chunk size is a latency budget
A streaming wrapper can accumulate a short chunk, run recognition on an overlapping buffer, and compare successive hypotheses. Small chunks make updates arrive sooner but provide less context and repeat more computation. Large chunks improve amortization and context, but the earliest sound waits longer for a result. A strategy such as local agreement commits a prefix only after multiple successive hypotheses agree, reducing flicker at the cost of delay.
Even a model that runs in zero milliseconds cannot publish a word before the required sound has arrived. Add capture buffering, resampling, feature extraction, encoder work, decoder work, scheduling, network delay, and stability policy. End-to-end latency is their sum along the critical path. An impressive real-time factor for complete files does not by itself establish low time-to-first-caption for a live user.
Available complete words in this scheduler
Scheduling toy using the known synthesis intervals, with zero compute time. A real streaming recognizer also revises hypotheses and pays for inference.
At 400 ms chunks, our last speech at 2.13 s cannot be included until the 2.4 s processing boundary. With a 400 ms endpoint wait, finalization happens at 2.8 s: 670 ms after speech ends. Reducing the chunk interval to 100 ms moves that release to 2.6 s. These are scheduling delays alone. Real inference and hypothesis stabilization add further delay, and a recognizer may revise the noun while waiting for its release.
A voice activity detector, or VAD, estimates whether a region contains speech. It can avoid spending a dense encoder pass on a long silence and can help propose utterance boundaries. It does not tell you which words were spoken. A threshold that works for loud close-microphone speech may miss a quiet speaker, and trimming exactly at the detected boundary can remove a consonant. Systems commonly preserve some leading and trailing context.
Overlap helps keep boundary words intact. Without a reconciliation policy, however, the same word can appear twice in neighboring transcripts. The wrapper must compare text, timestamps, and stable prefixes, then decide which region owns each word. A pause is also not necessarily the end of a turn: someone may hesitate after We see a
. Endpointing too early makes the interface interrupt the speaker; waiting too long makes it feel unresponsive.
What changes in a speech-native language model?
A conventional voice assistant forms a cascade: speech recognition writes text, a language model answers in text, and speech synthesis turns that answer into audio. The transcript is a useful boundary for logging, editing and evaluation. It also discards information. We see a cat
spoken in delight, fear, or sarcasm can have the same text but imply different responses.
Speech-native systems can condition on learned audio representations and generate audio directly, sometimes together with text. An audio tokenizer may compress a waveform into codec tokens, preserving acoustic details beyond spelling. The system then has to model timing, prosody and speaker behavior as well as linguistic content. Moshi (2024), for example, models text and codec-token speech streams and supports full-duplex dialogue. Its architecture is an example of this direction, not a claim that every speech model uses the same tokenizer or objective.
Audio → ASR → “We see a cat.”
Text LLM → answer text → speech synthesizer
Audio features or codec tokens → joint model
Time-aligned text + generated audio streams
Streaming ASR and speech-native dialogue therefore solve overlapping but different problems. A good transcription engine may intentionally normalize away tone and disfluency. A conversational audio model may need to preserve them and handle interruptions while its own speech is still playing. Both begin with sampled pressure changes, but their desired outputs determine which information they can afford to discard.
Pay for the listening
The physical signal has become tensors and conditional probabilities. Running those tensors has a cost in arithmetic, bytes, and sequential dependencies. The encoder and decoder stress the hardware differently. An encoder layer processes all 1,500 positions together, reusing each loaded weight across a large matrix multiplication. A cached decoder step processes one new text position per sequence and often spends much of its time moving weights and cached features.
Whisper’s published sizes range from tiny to large. Width, depth and vocabulary contribute differently to parameter count. The text embedding table alone has roughly vocabulary × width parameters and is also used as the final vocabulary projection. Large-v3 keeps the 32-layer encoder and decoder depth of the large family and increases mel channels to 128. Turbo retains the 32-layer, width-1,280 encoder but reduces the decoder to four layers.
| Model | Parameters | Width d | Encoder layers | Decoder layers | Mel channels |
|---|---|---|---|---|---|
| tiny | 39 M | 384 | 4 | 4 | 80 |
| base | 74 M | 512 | 6 | 6 | 80 |
| small | 244 M | 768 | 12 | 12 | 80 |
| medium | 769 M | 1024 | 24 | 24 | 80 |
| large-v3 | 1550 M | 1280 | 32 | 32 | 128 |
| large-v3-turbo | 809 M | 1280 | 32 | 4 | 128 |
The parameter counts come from OpenAI’s model table. The layer dimensions are recorded in the published model configurations, including large-v3 and turbo. Rounded counts are useful for comparing storage; they are not exact tensor-element totals for every language-specific variant.
Count a multiply and an add
A dense matrix product with dimensions m × k and k × n costs approximately 2mkn floating-point operations, counting each multiply-add as two. Ignore normalization, activation functions and biases for a first estimate. In an encoder block, attention projections contribute 8Nd2 FLOPs, the two feed-forward matrices contribute 16Nd2, and the two attention matrix products contribute 4N2d. Together that is:
Encoder block ≈ 24Nd2 + 4N2d
N = 1500 positions
Multiply by encoder depth and add the two convolutions. For small, with d = 768 and 12 encoder layers, this gives about 344 GFLOP for a 30-second window. Every position is available together, so the implementation can process many positions and multiple recordings in a batch. Dense work can be parallelized well, although its actual speed still depends on kernel quality, precision, device utilization and attention implementation.
The decoder adds cross-attention to its causal text attention. Cached self-attention retains the previous text keys and values. Cross-attention can project the encoder’s 1,500 vectors into keys and values once per decoder layer, then reuse them at every output step. That one-time setup costs approximately 4Nd2 per layer. Recomputing it for every new token would waste a large fraction of the work.
After that setup, one new token at text length t costs about 28d2 for the decoder layer’s projections and feed-forward matrices, plus 4(N + t)d for self- and cross-attention. The vocabulary projection adds roughly 2dV across the model, where V is vocabulary size. Longer transcripts therefore add sequential decoder passes even when the acoustic window length is unchanged.
Dense multiply-adds count as 2 FLOPs. Includes convolutions, projections, attention, MLP and vocabulary output; omits elementwise operations, prompt prefill, search and data movement.
Why the decoder waits for memory
At batch one, many decoder weight matrices are applied to one new vector. Each 16-bit weight supplies roughly one multiply and one add for two bytes fetched: about one FLOP per byte before reuse from caches. A large encoder matrix multiplication can use the same weight across many time positions. The accelerator’s available arithmetic is therefore much easier to keep busy during encoding than during one-sequence token generation.
The decoder must also read its caches. Large-v3’s cross-attention keys and values contain 2 × 32 × 1500 × 1280 numbers per sequence. At two bytes per number that is 245,760,000 bytes, about 234 MiB. Turbo’s four decoder layers reduce that cross-attention cache to about 29 MiB. Text self-attention adds a cache that grows with generated length. These activations are separate from model weights and temporary workspace.
Reducing decoder depth thus attacks several costs at once: weight traffic per token, cross-attention cache size, setup work and per-token arithmetic. It leaves the large encoder cost in place. In the calculator, switching large-v3 to turbo keeps the encoder bar identical while shrinking the decoder bar. The speedup in a real application depends on how long the transcript is, how it batches, and which stage was dominating.
2 × 1500 × 1500 × 64 = 288,000,000 FLOPs.
16,384 FLOPs; 2048 B of Q + 1024 B of K; 512 B of FP32 accumulators.
Follow one tile, then one token
The small encoder has 12 heads, each 64 coordinates wide. For one head, QKT is a GEMM with M = N = 1500 and K = 64: 288 million FLOPs, or 3.456 billion across 12 heads in one layer. Each head’s Q and K occupy 192,000 bytes apiece in FP16. Materializing all 1500 × 1500 scores would write 4,500,000 bytes per head in FP16 or 9,000,000 in FP32. Fused attention keeps score tiles on chip and updates softmax/value accumulators instead of storing that full grid.
A 16 × 8 output tile reduces across 64 coordinates. Fourmma.sync.aligned.m16n8k16 operations can build it from FP16 inputs and FP32 accumulation, a shape documented in NVIDIA’sPTX ISA 8.3. The tile needs 16 × 64 × 2 = 2048 input bytes from Q and 64 × 8 × 2 = 1024 from K, with 16 × 8 × 4 = 512 bytes of register accumulators. Neighboring tiles reuse inputs through shared memory and caches. The edges mask unused rows or columns because 1500 is not divisible by either tile dimension; kernel tile shapes vary with hardware and library.
Now take small at batch one with 100 text positions in its self-attention cache. A cached decoder step uses 14d2 projection/MLP weights per layer: four self-attention matrices, two cross-attention matrices for Q/output, and eight d2 weights in the MLP. Across 12 layers plus the 51,865 × 768 vocabulary projection, that is 138,922,752 weights, or 277,845,504 FP16 bytes per token before hardware-cache reuse. Cross-attention K/V projection weights belong to the one-time setup, so they are excluded from this repeated read.
The same step attends to 55,296,000 bytes of cached audio K/V and 3,686,400 bytes of text K/V. Those reads plus the weights total336,827,904 bytes, about 321.2 MiB per token; the new text K/V adds another 36,864 bytes of writes. The corresponding dense products do about 336.8 million FLOPs, giving roughly one FLOP per byte in this no-cache-reuse traffic model. Norms, biases, temporary activations and cache policy add details, but the low reuse explains why one-sequence decode often waits for memory. Encoding presents many query rows together; batching decoder requests creates more rows that can share weight reads.
Lower precision reduces the bytes moved and can accelerate matrix multiplication, provided the kernels support it. Weight quantization introduces scales and rounding, while cache quantization changes the stored attention evidence. The right choice depends on measured transcript quality for the relevant languages and audio conditions. A theoretical reduction in weight bytes is a bandwidth opportunity, not a promised end-to-end speedup.
At the instruction level, dense kernels tile matrices so values loaded into fast local storage feed many multiply-accumulate operations. GPUs can execute these products on matrix units, while CPUs use vector instructions and cache blocking. A fused attention kernel avoids materializing the full attention-score matrix in device memory. The mathematical output remains a weighted sum of values, but the intermediate storage and traffic can differ dramatically.
The original 30-second PCM16 waveform occupies under a megabyte. A large model’s weights occupy gigabytes, and its intermediate activations add further traffic. Most of the hardware cost comes from repeatedly applying learned matrices to compressed acoustic evidence, rather than transporting the original sound. That is why shrinking an input audio file with a codec does little to reduce the transformer’s arithmetic after decoding.
Follow our sentence back up: pressure becomes voltage, voltage becomes clocked integer samples, samples become windowed frequency estimates, frequencies become normalized mel channels, and those channels become contextual vectors. Only then does the decoder choose written pieces. Each transformation preserves something useful and discards something else. Understanding what survives each boundary makes it easier to explain a wrong word, a delayed caption, or a GPU that is busy moving bytes instead of multiplying them.
Encoding amortizes weights across time positions. Decoding adds one dependent token at a time, so caches, weight traffic and output length shape the cost of turning sound into text.
End of Speech recognition · 9 levels, Idea to Machine
Where to next
Next in AI models · 06 of 06
Diffusion models, from noise to an image
Turn noise into structure with a real diffusion sampler, then follow an image through denoising, guidance, latent patches and the GPU arithmetic behind every step.
Practical notes
Transcribe audio locally with word timestamps
Use faster-whisper with word_timestamps and VAD, consume its segment generator, and preserve source-time offsets. Choose model size and compute type for your machine.