Web and networkingDive 09 of 12

Video streaming, down to the decoder

Press play on a ten-minute talk. Follow the first picture, the next segment and a sudden tunnel through HLS, DASH, adaptive bitrate, browser buffers and the silicon that turns bytes into motion.

Levels
9
Reading
38 min
Depth
Idea to Silicon
Published

A phone, a ten-minute talk, one tunnel

The signal drops. The talk keeps going.

At 5:00, bandwidth falls from 8 to 1.5 Mbit/s. The buffer buys time to switch.

Network capacity0–10 Mbit/s
Playable buffer0–30 s
Requested video bitrate0–5 Mbit/s
0:00 · press play10:01

5:50 · capacity 1.5 Mbit/s · inspecting the tunnel

First picture 0.21 s · stalls 0.00 s · average 3.25 Mbit/s

Sequential, constant-size video segments; 60 ms per request; 30 s buffer cap; one segment to start. Dashed line: 5:00 tunnel. Zero requested bitrate: idle. The trace is available capacity, with no TCP ramp, audio or decode delay.

The EWMA + buffer cap run is plotted. All axes are linear and start at zero.

Level 01/09Idea

Start before the download ends

You board a train, open a ten-minute conference talk and press play. The speaker appears almost immediately. Five minutes later, the train enters a tunnel. The picture becomes softer, but the sentence continues. Several systems just cooperated: an encoder prepared alternative pictures, a server published a map, a player made a prediction, and a buffer covered the time that prediction took to change.

Keep one example in mind. The talk has five video versions, from 240p at 0.3 Mbit/s to 1080p at 5 Mbit/s. Your phone initially has 8 Mbit/s of available capacity. At wall-clock time 5:00, that falls to 1.5 Mbit/s. We will use four-second segments unless you change the controls. Audio is another track; the numerical model below budgets video alone so you can follow each multiplication.

Playing a file is already streaming

A browser can start playing an ordinary MP4 while the rest downloads. This is progressive download: one encoded timeline, usually one resource, read incrementally. Put the metadata near the beginning so the player learns where the samples live without fetching the whole file. HTTP range requests let it seek to another byte region. A ten-minute download does not have to finish before the first frame can appear.

The difficulty is that one file commits to one quality. A 5 Mbit/s encoding consumes 375 MB over 600 seconds, before audio and container overhead. On the open track, an 8 Mbit/s connection can fetch it faster than playback consumes it. In the tunnel, each second of video takes over three seconds to download. A large buffer postpones the stall; it cannot make that arithmetic sustainable.

A delivery method that offers interchangeable encodings of a timeline and lets the player choose which one to request next. turns that one commitment into a series of smaller decisions. The publisher cuts each version into aligned pieces and lists their locations in a manifest. The player might request the first piece at 240p, move to 1080p once it has evidence about the network, then request a later piece at 480p. It downloads only the selected pieces, not all five versions.

Each request must buy enough playback time to cover the time spent downloading it. At tunnel capacity, the lower rendition changes the arithmetic.

The buffer is time you have already paid for

A playback buffer holds media that arrived ahead of the playhead. Measure it in playable seconds, not just bytes. Twelve megabytes can mean roughly 19 seconds of 5 Mbit/s video or 80 seconds at 1.2 Mbit/s. The player needs the duration because its deadline is the next missing picture’s presentation time. A byte count alone says little about how long the speaker can keep talking.

There are three competing goals. Start quickly, show a good picture, and avoid interruption. Waiting for thirty seconds of video improves resilience but delays the first picture. Starting with one small low-quality segment improves startup while limiting initial detail. A higher bitrate spends the buffer faster during a fade. For live video there is a fourth constraint: every second deliberately buffered also moves you farther behind the camera.

These pieces are ordinary HTTP objects. They can use existing authentication, congestion control and CDN infrastructure. The server generally answers requests for named objects; it does not need to keep a custom streaming session for every viewer. A CDN can serve the same segment bytes to thousands of phones whose buffers and quality choices differ. The player owns the short-term scheduling decision.

Under a second is a budget, not a promise made by the format. In our warm-path model, the first 240p four-second segment contains 1.2 Mbit. At 8 Mbit/s, payload transfer takes 150 ms. Add the model’s 60 ms request cost and it becomes 210 ms. A real startup must also obtain the manifest and initialization data, establish any cold connections, initialize the decoder and meet autoplay rules. A key or license exchange may add another dependency. Small first requests make room for these costs; naming the protocol does not remove them.

DeliveryWhat the client choosesWhat happens in the tunnel
Progressive MP4One encoding and byte rangesConsumes the encoding’s fixed average rate
HLS / DASHRendition and next segmentCan trade picture detail for a sustainable rate
Real-time transportSender and receiver coordinate a live flowMay drop detail or frames to preserve deadlines

The buffer buys time to change your next request. It cannot make an unsustainable bitrate sustainable.

Level 02/09Interface

Encode five ways to tell the same story

The player can only choose alternatives the publisher created. The five versions form a A set of encoded renditions ordered by their bitrate and usually their resolution.. Each rung trades detail, artifacts, decoding work and storage against the number of bits required per second. Resolution is only one coordinate: a clean 720p encode can look better than an aggressively compressed 1080p encode.

Our 240p, 360p, 480p, 720p and 1080p versions use 0.3, 0.7, 1.2, 2.5 and 5 Mbit/s. Those are chosen teaching rates, not a universal recommendation. A mostly static conference slide and a handheld shot of a crowd have different rate–distortion curves. The lower rate often preserves the slide well until tiny text disappears; the crowd keeps generating motion and texture that need residual bits.

A ladder should fit the pictures

In per-title encoding, the publisher tries candidate resolutions and quality settings, measures the results, then keeps useful trade-offs. A point is dominated when another encoding is both smaller and at least as good under the chosen quality measure. Removing such points saves storage and avoids offering choices that spend more bandwidth for no benefit. A metric is evidence, however: text, animation, grain and skin can expose its blind spots. Human review still matters.

The package also needs compatible time boundaries. Imagine switching at exactly 5:04 from one rendition to another. If the first picture in the new segment depends on a reference picture that only exists in the old rendition’s decoder state, the switch cannot decode correctly. Encoders therefore coordinate random access points, timestamps and segment boundaries across a switching set. Merely chopping five MP4 files into equal byte lengths accomplishes none of this.

A A group of pictures with a configured prediction structure and interval between random access pictures. is the frame structure between access points. A closed, independently decodable GOP at a segment start is a straightforward way to permit switching. Open GOPs and leading pictures need more careful signaling and player support. The word “keyframe” is often used loosely; the real requirement is that the specified access point allows the required decoding operation.

426 × 240
0.3 Mbit/s
640 × 360
0.7 Mbit/s
854 × 480
1.2 Mbit/s
1280 × 720
2.5 Mbit/s
1920 × 1080
5.0 Mbit/s
240p
● 0 s● 4 s● 8 s
360p
● 0 s● 4 s● 8 s
480p
● 0 s● 4 s● 8 s
720p
● 0 s● 4 s● 8 s
1080p
● 0 s● 4 s● 8 s

4 s × 1.2 Mbit/s = 4.8 Mbit = 0.60 MB per video segment. 150 segments in the talk.

Bars use a shared 0–5 Mbit/s scale. Dots mark independently decodable segment starts; the spacing shows time, not byte size. Decimal MB; audio and overhead excluded.

480p is sustainable at 1.5 Mbit/s in this constant-rate model, with 0.3 Mbit/s left for overhead and other tracks.

The ladder controls request size; aligned boundaries control when a change can take effect. Try a longer segment at 1080p, then compare its tunnel download time with the duration it supplies.

Why not make every segment tiny?

Two to six seconds is a useful conventional range to explore, rather than a rule. Shorter segments give more frequent switching opportunities and smaller individual mistakes. They also create more HTTP requests and packaging metadata. More frequent independently coded pictures can reduce coding efficiency. A four-second piece of 1080p video averages 2.5 MB in our model; a six-second one averages 3.75 MB. If you choose poorly just before the fade, you are committed for longer.

Longer segments amortize fixed costs and can give the encoder more room for efficient prediction. But “segment” need not mean “wait until every byte exists.” Low-latency packaging can expose smaller parts of the same segment. We will separate those concepts later. At this stage, the important contract is that each rendition has a compatible media timeline and the manifest gives enough information to address it.

Common Media Application Format: an ISO format that constrains fragmented ISO BMFF media for interoperable segmented delivery. helps two manifest ecosystems share the same media objects. You can describe compatible CMAF fragments using either an HLS playlist or a DASH MPD. This can save duplicate packaging, storage and cache entries. It does not make the manifests interchangeable or guarantee that every codec, encryption scheme and client combination works. The governing specification is ISO/IEC 23000-19; the browser’s ISO BMFF byte-stream note, §3 gives a useful public view of the initialization contract.

Two descriptions can refer to one compatible set of CMAF objects. Packaging reuse happens below the manifest.
Check yourselfYou cut all five renditions at 4, 8 and 12 seconds. Is switching automatically safe?

Level 03/09Interface

A menu, then a shopping list

The player now needs two kinds of knowledge: what versions exist, and where their next media pieces live. HLS expresses these as playlists. A multivariant playlist offers variant streams, often combining a video rendition with a selectable audio group. A media playlist lists the segments of one media timeline. An audio-only rendition has its own media playlist too.

Read HLS as a grammar

The first line is #EXTM3U. A #EXT-X-STREAM-INFline contains an attribute list, and its following URI identifies the variant’s playlist. BANDWIDTH is the peak segment bitrate for the playable combination described by the variant; AVERAGE-BANDWIDTH describes its average. These are bits per second. A player must budget all simultaneously selected media, rather than forgetting audio when it reads the video resolution.

RESOLUTION=854x480 is a pair of decimal dimensions. CODECS="avc1.64001f,mp4a.40.2" is one quoted attribute value containing a comma. Splitting the whole line on commas would turn two codecs into an invented attribute. HLS quoted strings have their own rules; JSON backslash escaping is not part of that grammar. Attribute names are case-sensitive and duplicate names are invalid. See RFC 8216 §4.2 and §4.3.4.2.

Inside the media playlist, EXTINF gives the next segment’s duration.EXT-X-TARGETDURATION bounds those durations after rounding to the nearest integer; it is not a promise that every segment lasts exactly that long. EXT-X-MEDIA-SEQUENCE numbers the first listed segment and increments for subsequent ones. With a sliding live window, the first listed number rises as old entries leave. It is not a timestamp and it does not reset just because a client reloads the playlist.

EXT-X-MAP identifies the initialization section for the following fragmented MP4 media. EXT-X-ENDLIST means no more media segments will be added. Our VOD sample lists all 150 four-second segments of the talk. Try changing one duration to 4.49 and then 4.5 with target duration 4: the nearest-integer rule makes the second change invalid. These details come from §4.3.2.1, §4.3.2.5 and §4.3.3.1.

#EXTM3U

Playlist signature; must be the first line.

#EXT-X-VERSION:7

Minimum compatibility version needed to read this playlist.

#EXT-X-INDEPENDENT-SEGMENTS

Every media segment can be decoded without other segments.

#EXT-X-STREAM-INF:BANDWIDTH=428000,AVERAGE-BANDWIDTH=398000,RESOLUTION=426x240,CODECS="avc1.64001f,mp4a.40.2"

Variant attributes; the next URI identifies its media playlist.

240p/index.m3u8

URI of this variant’s media playlist.

#EXT-X-STREAM-INF:BANDWIDTH=828000,AVERAGE-BANDWIDTH=758000,RESOLUTION=640x360,CODECS="avc1.64001f,mp4a.40.2"

Variant attributes; the next URI identifies its media playlist.

360p/index.m3u8

URI of this variant’s media playlist.

#EXT-X-STREAM-INF:BANDWIDTH=1328000,AVERAGE-BANDWIDTH=1208000,RESOLUTION=854x480,CODECS="avc1.64001f,mp4a.40.2"

Variant attributes; the next URI identifies its media playlist.

480p/index.m3u8

URI of this variant’s media playlist.

#EXT-X-STREAM-INF:BANDWIDTH=2628000,AVERAGE-BANDWIDTH=2378000,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"

Variant attributes; the next URI identifies its media playlist.

720p/index.m3u8

URI of this variant’s media playlist.

#EXT-X-STREAM-INF:BANDWIDTH=5128000,AVERAGE-BANDWIDTH=4628000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"

Variant attributes; the next URI identifies its media playlist.

1080p/index.m3u8

URI of this variant’s media playlist.

240p/index.m3u8

BANDWIDTH = 428000; AVERAGE-BANDWIDTH = 398000; RESOLUTION = 426x240; CODECS = avc1.64001f,mp4a.40.2

360p/index.m3u8

BANDWIDTH = 828000; AVERAGE-BANDWIDTH = 758000; RESOLUTION = 640x360; CODECS = avc1.64001f,mp4a.40.2

480p/index.m3u8

BANDWIDTH = 1328000; AVERAGE-BANDWIDTH = 1208000; RESOLUTION = 854x480; CODECS = avc1.64001f,mp4a.40.2

720p/index.m3u8

BANDWIDTH = 2628000; AVERAGE-BANDWIDTH = 2378000; RESOLUTION = 1280x720; CODECS = avc1.64001f,mp4a.40.2

1080p/index.m3u8

BANDWIDTH = 5128000; AVERAGE-BANDWIDTH = 4628000; RESOLUTION = 1920x1080; CODECS = avc1.640028,mp4a.40.2

Multivariant playlist: five variant references. URI text is inspected locally; nothing is fetched.

HLS: core and displayed low-latency tags are validated; unsupported attributes and other extension tags are retained and marked. Reference list shows full segments; parts and hints are explained alongside their source lines. DASH: static SegmentTemplate addressing, one BaseURL per level, first three segment references per representation. Limits: 32 KiB, 512 XML elements, 1,024 HLS lines.

Inspect the talk’s HLS and DASH manifests, or the bundled Apple and Envivio samples. Edit the source, break a field, and inspect again; every displayed URI remains text.

DASH describes a presentation tree

A DASH MPD is XML. A Period describes an interval of the presentation. Within it, an AdaptationSet groups media alternatives, such as video versions with a common role. A Representation is one concrete encoding. Audio and subtitles usually occupy different adaptation sets, so choosing a video representation is only part of assembling the presentation.

SegmentTemplate avoids listing hundreds of almost identical URLs. In our number-addressed sample, $RepresentationID$ selects the rendition’s directory and $Number%03d$ formats segment numbers such as 001. startNumber sets the first index. duration="4000" and timescale="1000" mean four seconds, not four thousand seconds. Templates inherit down the MPD tree; a representation can override selected fields without repeating everything.

A SegmentTimeline instead describes runs in sample-time ticks.S t="0" d="4000" r="149" means one interval followed by 149 repetitions: 150 segments total. With $Time$, the URL receives the start tick, so the first names contain 0, 4000 and 8000. An omitted t continues the previous run. A negative-one repeat runs to the next explicit start or the period boundary; blindly expanding it as an infinite array would be a parser bug.

URL time and presentation time have different origins. With timescale 1000, duration 4000 and presentation-time offset 900, duration-based $Time$ names are 900.m4s, 4900.m4s and 8900.m4s. Their nominal presentation starts are 0, 4 and 8 seconds after the Period starts. The mapping is Period start + (URL time − presentation-time offset) / timescale; explicit SegmentTimeline values already supply their sample-time positions. See DASH-IF §18.4. The inspector shows both coordinates separately.

The current catalog edition is ISO/IEC 23009-1:2026, edition 6. The public DASH-IF timing model, §9.1 explains sample time and its mapping to presentation time; §9.2 covers segment references. A Period start, presentation-time offset and sample timescale are distinct quantities. Mixing them can generate plausible-looking URLs for the wrong pictures.

HLS’s published baseline remains the 2017 RFC. As checked for 15 November 2026, HTTP Live Streaming 2nd Edition revision 22 is in RFC Editor processing, not yet a replacement RFC. Its text is dated 1 May 2026 and carries a 2 November expiry date; the processing history matters more than treating that date as a new protocol version. We use its low-latency tags later and name the draft explicitly.

A manifest describes choices and time. The player turns that description into requests.

Level 04/09Runtime

Put time into the browser

The manifest is not playable media. A JavaScript player parses it, chooses a representation, requests bytes, and hands those bytes to the browser. In a Media Source Extensions pipeline, MediaSourceattaches a programmable source to a media element; one or more SourceBuffer objects accept encoded media. The browser retains responsibility for parsing the media byte stream, decoding, synchronizing tracks and presenting pictures.

Native HLS playback can hide much of this inside the platform. A page that uses MSE performs more of the manifest and download work itself. MSE is a timed-media ingestion API, not an HLS or DASH interpreter. Giving appendBuffer() a playlist does not make the browser fetch its segments.

One append at a time

Wait for sourceopen, choose a supported MIME type with a codec string, then create the SourceBuffer. Append initialization data before dependent fragments. A successful call starts an asynchronous operation and sets updating. The next append or removal must wait until that operation finishes. The updateend event closes the operation, including operations that ended through error or abort; success needs its own error handling.

A synchronous QuotaExceededError means the append could not start because buffer space is insufficient. Clear appropriate played ranges or reduce future buffering, then retry deliberately. A synchronous InvalidStateError can mean the buffer is already updating, has been removed, or the media element has an error. An asynchronous decode or append error follows a different path. Catching only the call cannot catch events that arrive later. These transitions are defined in MSE’s appendBuffer algorithm and its prepare append steps.

SourceBuffer model · simulated events, no media API calls

Manifest + ABR
Page JavaScript
appendBuffer
Encoded samples
Decode + present
Browser and hardware
0 s12 s24 s

readyState: open · initialized: no · updating: false

currentTime: 0 s · buffered: empty · resident: 0 / 12 s

sourceopen: MediaSource is ready for a SourceBuffer.

Append an initialization segment, then append media. Twelve resident seconds is this model’s quota; played bytes stay resident until removed.

Drive the buffer model through initialization, append, playback and eviction. Fill its quota, play ahead, remove the played ranges, then retry an append.

Ahead, behind, and missing

The model makes something easy to miss visible: playing a sample does not immediately free its storage. Browsers may retain a back buffer for seeking. Eviction can reclaim old coded frames under implementation-specific policies. Your scheduler also needs limits on future buffering, particularly when high-resolution samples occupy substantial memory. A duration budget and a byte budget solve different parts of that problem.

buffered exposes ranges of media time. To estimate forward buffer, locate the range containing currentTime and subtract the playhead from that range’s end. Do not subtract it from the last range’s end if there is a gap in between. With separate audio and video buffers, usable playback also depends on their overlap. Twenty seconds of video with only one second of audio is not twenty seconds of synchronized talk.

Seeking changes the scheduling problem. The player should invalidate requests for media it no longer intends to append, arrange the correct initialization context, and fill the new region. A late completion from the old seek must not silently overwrite the new state. A generation counter or cancellation signal often makes this ownership explicit. The same principle applies when a reader edits an inspector on this page: old results immediately stop claiming to describe the new input.

Managed Media Source gives the user agent more control over buffering and resource use, with streaming signals and buffered-change notifications. WebKit introduced it on iPhone in Safari 17.1 on iOS 17.1; the launch requirement was an AirPlay-compatible alternative source or explicitly disabling remote playback. Feature detection and platform testing remain necessary. That launch fact is narrower than promising ordinary MSE behavior on every iPhone browser.

Encryption adds a parallel conversation

With protected media, the page also uses Encrypted Media Extensions to work with a content decryption module, or CDM. Initialization data leads to a key-system session; the application relays the session’s license message to its license service and returns the response. The CDM decides whether and how keys can be used. EME defines the API between these pieces, not one universal DRM protocol.

Widevine, FairPlay and PlayReady are key-system families, each with platform and robustness constraints. Common Encryption signaling can describe encrypted samples across systems; cenc and cbcs name different protection schemes, based on counter mode and CBC pattern encryption respectively. Packaging, encryption scheme and device support must agree. A valid manifest and enough bandwidth cannot compensate for a missing supported decoder or usable key.

Check yourselfA SourceBuffer has ranges [0, 12] and [20, 30]. The playhead is at 10. How much continuous media lies ahead?

Level 05/09Runtime

Choose the next four seconds

You have a menu and a way to append bytes. What should the next request buy? Adaptive bitrate selection: the client’s policy for choosing a representation as conditions change. turns noisy observations into a decision with a deadline. It cannot observe the future tunnel. At best, it knows how previous transfers behaved, how much continuous media is buffered, and which segment choices are available.

For a constant-rate segment, size is bitrate multiplied by duration. Download time is size divided by available throughput, plus request costs. If a four-second 1080p segment needs 13.33 seconds of payload transfer in the tunnel, it drains 9.33 seconds of the buffer before paying four back. A buffer of twenty seconds can absorb one such surprise. Repeating it guarantees a stall.

The recurrence after a completed segment is simple: subtract time spent waiting or downloading, floor at zero, then add the received segment’s duration. Any uncovered time after playback starts is rebuffering. Startup waiting is a separate measure. A useful simulator must preserve these distinctions; calling all idle time “buffering” would obscure the trade you are trying to inspect.

Bnext = max(0, B − tdownload) + p
stall = max(0, tdownload − B)

Same network. Four different decisions.

Choose what the player knows

Choose a trace or edit its thirty-second intervals. The buffer buys time to switch.

Network capacity0–10 Mbit/s
Playable buffer0–30 s
Requested video bitrate0–5 Mbit/s
0:00 · press play10:01

5:50 · capacity 1.5 Mbit/s · inspecting the tunnel

Edit the trace · each value lasts 30 seconds, Mbit/s
AlgorithmStartupStall time / eventsSwitchesMean video rate
Throughput · 85%0.21 s0.00 s / 033.25 Mbit/s
BBA-00.21 s0.00 s / 0173.26 Mbit/s
BOLA-BASIC0.21 s0.00 s / 0643.24 Mbit/s
EWMA + buffer cap0.21 s0.00 s / 033.25 Mbit/s

Sequential, constant-size video segments; 60 ms per request; 30 s buffer cap; one segment to start. Dashed line: 5:00 tunnel. Zero requested bitrate: idle. The trace is available capacity, with no TCP ramp, audio or decode delay.

The EWMA + buffer cap run is plotted. All axes are linear and start at zero.

Every algorithm watches the same wall-clock capacity trace. Edit any thirty-second interval, change segment length, and scrub the plot to inspect the decisions around the tunnel.

Throughput: trust the last transfer

The first policy chooses the highest rung below 85% of the last measured payload throughput. It starts at the lowest rung because it has no measurement yet. This makes the decision easy to audit, and the safety margin leaves capacity for variation. It also makes a strong assumption: the last transfer predicts the next. A request started just before 5:00 can carry an old prediction into the tunnel.

A short, cached request can exaggerate the available long-term rate; a congested request can temporarily depress it. Transfer timing must distinguish request latency from payload movement. Production estimators add smoothing and minimum sampling rules. The minimum of fast and slow duration-weighted EWMAs is the central idea in hls.js’s bandwidth estimator: react quickly when conditions worsen and demand sustained evidence before trusting an improvement.

BBA-0: let the buffer vote

The baseline buffer-based policy follows the reservoir, cushion and hysteresis construction in Huang and colleagues, §3. Below five buffered seconds it selects the lowest rung. Above twenty-five it selects the highest. Between those points, a linear function maps occupancy to a target rate. Upward changes choose a rung strictly below that function; downward changes choose a rung strictly above it. The current rung stays put until the function crosses an adjacent rung, avoiding needless switching near a threshold.

Those five- and twenty-five-second bounds are our parameters, not the paper’s deployment settings. This is the baseline BBA-0 rule, without later startup and variable-size chunk extensions. It uses no throughput forecast. A shrinking buffer is its evidence of trouble, which means it can continue spending an existing reserve after a capacity drop. That behavior can be useful for a brief dip and costly during a sustained one.

BOLA-BASIC: price utility against occupancy

BOLA Algorithm 1 scores each segment choice by utility, buffer occupancy and size. Here utility is log(rate / lowest rate), occupancy Q is buffered duration divided by segment duration p, and segment size S is rate × p. We set γp = 5 and choose V so the thirty-second maximum satisfies the paper’s finite-buffer bound. A negative score for every rendition makes the policy wait rather than fetch more.

scorem = [V(vm + γp) − Q] / Sm
Q = B / p · vm = ln(rm / rmin)

The division by size matters. Picking the greatest numerator alone would always favor the highest utility whenever fetching was allowed. Buffer and utility also have different units; using seconds for Q while keeping V calibrated in chunks changes the algorithm. The comparison implements the basic fixed-parameter policy, rather than the paper’s additional finite-video and oscillation enhancements.

A teaching hybrid

The fourth policy uses the minimum of bias-corrected, duration-weighted EWMAs with three- and nine-second half-lives. It selects below 65% of that estimate when the buffer is under eight seconds and below 85% otherwise. These coefficients define this page’s teaching hybrid. A full hls.js controller also considers loading deadlines, errors, restrictions and other state; dash.js DYNAMIC switches between throughput and buffer-based selection. Neither production player is reproduced by one estimator.

Read all four outcomes together. Mean bitrate rewards quality spending but hides the visual cost of switching and interruption. Zero stalls under the tunnel trace does not establish safety under a deeper fade. The deep-fade preset drops below even the smallest video rung; eventually every policy must consume its reserve. A real player might abandon an in-flight request, choose audio-only operation, or pause. This model finishes each request so the cost of the original decision remains visible.

ABR chooses the next request using delayed evidence. Segment duration determines how expensive a wrong prediction becomes.

Level 06/09System

Live means staying behind

The train example is on demand: every picture already exists. Now imagine the same speaker is talking live. At the encoder, the future has not happened yet. The player must stay far enough behind the camera that pictures can be encoded, published, downloaded and decoded before their presentation deadlines. Even an infinitely fast network cannot deliver a picture that has not been captured.

Conventional segment publication often waits until a whole segment is ready. If that segment lasts four seconds, a picture near its beginning can spend much of those four seconds waiting for publication. The player then deliberately keeps more media ahead of its playhead to survive jitter. Glass-to-glass latency includes capture and encoding, publication, delivery, playback reserve, decoding and display scheduling. Reducing only transfer time attacks one part of the sum.

Publish part of a segment

Low-Latency HLS exposes partial segments using EXT-X-PART, with a target part duration signaled by EXT-X-PART-INF. A EXT-X-PRELOAD-HINT can identify an upcoming object before it is fully available. Blocking playlist reload lets a client ask for an update containing a specified media sequence and part rather than repeatedly polling an unchanged playlist. These mechanisms are described in 2nd Edition draft §4.4.3.7, §4.4.4.9 and §6.2.5.2.

A partial segment is not automatically an independent random access point. It can contain pictures that depend on earlier parts of the segment. The player can fetch and decode incrementally while still switching or joining only at suitable boundaries. This lets publication granularity become smaller without requiring a large independently coded picture every two hundred milliseconds.

Live talk · 480p/index.m3u8
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:75
#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.500,HOLD-BACK=12.000
#EXT-X-PART-INF:PART-TARGET=0.500
#EXT-X-MAP:URI="init.mp4"
#EXTINF:4.000,
seg-075.m4s
#EXTINF:4.000,
seg-076.m4s
#EXTINF:4.000,
seg-077.m4s
#EXT-X-PART:DURATION=0.500,URI="seg-078.part-0.m4s",INDEPENDENT=YES
#EXT-X-PART:DURATION=0.500,URI="seg-078.part-1.m4s"
#EXT-X-PRELOAD-HINT:TYPE=PART,URI="seg-078.part-2.m4s"

This complete live-playlist example has three four-second segments, then the first two half-second parts of segment 78. PART-HOLD-BACK=1.500 recommends staying three parts behind the live edge. The preload hint names part 2 before publication completes; it does not make that part a published playlist entry.

After seeing part 1, the client can make this blocking reload request. Both query values identify the update it wants: sequence 78 and part index 2. The server can defer its answer until that part or a later update is available, or return an error under the draft’s rules. Reloading the playlist and requesting the hinted media object are separate HTTP operations.

Blocking playlist reload · request text only
GET /live/talk/480p/index.m3u8?_HLS_msn=78&_HLS_part=2 HTTP/1.1
Host: talk.example
Publication wait0.25 s
Playback reserve1.50 s
Transfer + request0.46 s
Decode + display0.05 s

Illustrative glass-to-glass budget: 2.26 seconds.

0–32 s common scale. Uniform capture phase gives half a publication unit of waiting; reserve is units × unit length; video is 1.2 Mbit/s over 1.5 Mbit/s capacity, plus 60 ms and 50 ms decode/display. Encoder delay omitted.

Three half-second parts give a 1.5 s playback reserve. Less reserve leaves less time to survive a fade.

A small publication unit removes waiting, but a small playback reserve also removes protection. Change both and inspect a simple live latency budget.

LL-DASH commonly uses chunked CMAF delivery: a segment response can yield completed movie fragments while the encoder is still producing later chunks. The MPD describes when requests may be made and whether the response is complete at availability time. HTTP/2 and HTTP/3 carry incremental data without HTTP/1.1’s chunked transfer coding; “chunked CMAF” names media production granularity, not a requirement to use that HTTP/1.1 coding. The DASH-IF availability model explains how availability offsets and timing interact.

A CDN in this path has to forward growing responses promptly. An origin can produce half-second chunks perfectly while an intermediate proxy waits for the entire four-second response. A low latency player then receives four-second bursts and cannot recover the publication time already lost. Origin, CDN, manifest timing and client scheduling are one timing system, even though they run in different organizations.

When a small reserve is still too much

For a video call, several seconds of reserve destroy turn-taking. WebRTC uses a real-time media stack with jitter buffering, congestion feedback and loss handling designed around deadlines. It usually gives up some quality or completeness rather than insisting that every old media byte arrive. WHEP describes an HTTP signaling approach for WebRTC egress; the media itself still uses WebRTC transport. As checked on 15 November 2026,draft-ietf-wish-whep-05is an active working-group Internet-Draft intended for Proposed Standard, submitted to the IESG and in “AD Evaluation::Revised I-D Needed”. It has not been published as an RFC.

Media over QUIC explores a different delivery structure: named media objects, subscriptions, relays and prioritization over QUIC or WebTransport. Independent streams can keep a late object from blocking unrelated newer objects. That does not eliminate radio capacity limits or make codec dependencies disappear. A disposable picture and a needed reference picture have different consequences when dropped.

As checked for 15 November 2026, MoQ Transport revision 23 is still an active working-group Internet-Draft. Its object and subscription vocabulary is useful for understanding the design; interoperability claims need a named revision. It should not be presented as a universally deployed replacement for HLS, DASH or WebRTC. These families optimize different combinations of audience scale, delay, cacheability and interaction.

ApproachUseful unitLatency pressureDistribution
Conventional HLS / DASHCompleted segmentSeconds of publication and reserveHTTP caches
LL-HLS / LL-DASHPart / movie fragmentSmaller publication units and reserveHTTP infrastructure that forwards promptly
WebRTC / WHEP egressReal-time media packetsConversation and interaction deadlinesReal-time relays / servers
MoQ draftNamed media objectsObject scheduling and expirationQUIC-aware relays

Level 07/09Machine

Open the segment, box by box

Follow one requested .m4s object below HTTP. The filename extension is a convention; the bytes establish the format. Fragmented MP4 uses the ISO base media file format, specified by ISO/IEC 14496-12, edition 8 (2026). Its fundamental structure is a sequence of boxes. A box says how many bytes belong to it, what type it has, and what its payload means.

A normal box begins with a four-byte unsigned big-endian size and a four-byte type. The size includes the header. The bytes 00 00 00 08 66 72 65 65 describe an empty eight-byte free box. A size of one introduces an eight-byte extended size after the type. A size of zero extends to the end of the file. A uuid box also carries a sixteen-byte extended type. Treating every box as exactly eight header bytes misaligns the parser immediately.

Initialization tells you how to read the samples

The initialization segment provides ftyp and moov. The first advertises file compatibility; the second describes the movie’s tracks and sample configurations. A trak contains track metadata; its mdia describes media timing and the handler, with lower-level tables inside. Video sample descriptions eventually identify a codec configuration such as AVC parameter sets. Those bytes explain how to interpret future compressed samples without carrying that information in every HTTP object.

In a fragmented presentation, the initialization metadata does not need a complete table of every sample in the whole talk. The fragment supplies changing timing, sizes and flags. MSE’s public ISO BMFF byte-stream format §3 defines the initialization shape and the fragment-related constraints expected by the browser.

ftyp · file type

Offset 0. Size 28 bytes, including its 8-byte header.

Major brand: iso5. The compatible brands describe acceptable readers.

00 00 00 1c 66 74 79 70 69 73 6f 35 00 00 02 00 69 73 6f 35 69 73 6f 36 6d 70 34 31

Real encoder output: 2,352 bytes, four H.264 samples. Select a box to inspect its exact span.

64 KiB, 512 boxes, 16 nesting levels. Known container boxes recurse; codec sample descriptions and media payload remain opaque. No media is downloaded or played.

Real libx264 and FFmpeg bytes expose each boundary. Load init or media separately, select tfdt or trun, and inspect sizes, flags and offsets.

A fragment tells you where and when

A media segment has a moof containing fragment metadata, followed by media data in mdat. A segment may contain multiple such pairs. mfhd supplies the movie-fragment sequence number. A track fragment traf contains tfhd, which identifies the track and can set defaults; tfdt, which supplies a base decode time; and one or more trun boxes describing runs of samples. Optional fields are controlled by flags, not by a fixed row of guessed offsets.

A trun can specify per-sample duration, size, flags and composition-time offsets. Fields omitted there may come from the track-fragment or track-extends defaults. Its data offset, when present, is signed and interpreted relative to the applicable base. The surrounding metadata is necessary to find a sample. Searching the file for the literal letters mdat or assuming the payload starts at a universal byte number is not parsing a container.

Follow the bundled fixture all the way into its samples. In the complete file, moof starts at byte 770. The track fragment sets default-base-is-moof, and trun.data_offset is 128, so the first sample starts at 770 + 128 = 898. That is exactly the mdat payload start, after its eight-byte header. The four sample sizes are 1,155, 94, 74 and 64 bytes: 1,387 in total. They occupy [898, 2053), [2053, 2147), [2147, 2221) and [2221, 2285), ending exactly at mdat’s boundary. The remaining file bytes belong to the fragment index, not another sample.

Decode time and presentation time can differ. Reference pictures must exist before dependent pictures decode, but display order can place those dependents earlier. The time base is also a track property: the fixture’s 16,384 ticks per second means each 8,192-tick sample lasts half a second. Its four samples occupy two seconds. This tiny fixture is a structural microscope, separate from the ten-minute numerical example; it uses constrained-baseline AVC and no B pictures to keep its bytes small.

The parser walks known container boundaries and keeps unknown payloads opaque. It checks each size against the containing span before reading fields. Extended lengths are read as 64-bit integers before conversion to a JavaScript number. Depth, box count and total byte limits keep hostile input bounded. A decoder may have many more semantic constraints; a well-formed box tree is the first layer of validation, not proof that its sample bytes form a supported picture.

A codec identifier is a small compatibility contract

avc1.64001f starts with an AVC sample-entry name. Its suffix consists of three hexadecimal bytes: profile 0x64, compatibility 0x00 and level 0x1f. These decode to High profile, compatibility flags zero and level 3.1. They are not three decimal numbers and “64” is not the resolution. RFC 6381 §3.3 specifies this mapping. Level 1b needs special handling: Baseline, Main and Extended use level_idc 11 with constraint_set3_flag set, while High-family profiles use level_idc 9. Reserved or unsupported level bytes are labeled rather than converted into invented level numbers.

The suffix grammar depends on the codec family. mp4a.40.2 identifies MPEG-4 Audio with object type indication 0x40 and audio object type 2, AAC Low Complexity. HEVC identifiers such as hvc1 and AV1’s av01 use their own extended forms. A browser capability query needs the actual complete type and configuration. The talk’s manifest uses AVC level 4 for its 1080p, 30-frame/s rendition and level 3.1 for the smaller versions. Its HLS bandwidth totals add a 128 kbit/s AAC audio budget; the simulator continues to model video alone.

High profile (100); compatibility 0x00; level 3.1.

Change the AVC profile, compatibility or level bytes. HEVC and AV1 use different suffix grammars, so the inspector explains their family instead of interpreting them as AVC.
Check yourselfA BMFF box declares size 16 and uses the normal eight-byte header. How much payload belongs to it?

Level 08/09Machine

Spend bits only on what changed

Open mdat and you still do not have an array of display pixels. You have a compressed description of pictures. The encoder searches for a cheap way to describe them; the decoder follows a specified reconstruction procedure. The two perform very different amounts of work. An encoder may try hundreds of candidate decisions that the decoder later executes just once.

Start with one luma block from a picture of the speaker. Nearby samples are correlated. Across frames, the background often barely changes, while the speaker’s hand moves. Video compression uses both facts: predict a block from neighboring reconstructed samples or reference pictures, then encode the remaining difference, called the residual. Good prediction makes a complicated scene produce a comparatively simple residual.

I, P and B describe dependencies

Intra-coded pictures use spatial prediction within the picture. Predictive pictures can use earlier decoded reference pictures, with motion information locating useful regions. Bi-predictive structures can combine references that appear before and after the current picture in display order. These are useful introductory categories; actual codecs allow block-level decisions, multiple references and more complicated structures.

A motion vector says where a prediction comes from, often with fractional-sample precision. The decoder interpolates reference samples as specified, adds the decoded residual, and applies the codec’s reconstruction filters. Even a “copy from the previous frame” decision is carefully defined: which reference surface, which position, which interpolation and which boundary behavior all matter for deterministic output.

Display order: I0, B1, B2, P3 · one illustrative prediction structure

Decode order: I0, P3, B1, B2

Predicted block+Decoded residual=Reconstructed block
Reference dependencies explain both decoding order and startup costs. This B-picture example requires P₃ to decode before B₁ and B₂ can be displayed.

Turn a block into frequencies

A transform represents a block with basis patterns. The lowest-frequency coefficient captures the average; higher-frequency coefficients capture increasingly rapid changes in each direction. A smooth block often concentrates much of its energy into a few coefficients. That rearrangement is reversible before rounding: the transform itself does not magically delete information.

The interactive below takes a real decoded 8 × 8 image block, subtracts 128 and computes an orthonormal discrete cosine transform. For each frequency pair (u, v), it projects the samples onto a cosine basis. Modern video codecs use specified transform families and integer approximations, often on several block sizes. The floating-point DCT here isolates the core idea so you can watch the coefficients directly.

qu,v = round(Cu,v / Δ)
Ĉu,v = qu,v · Δ
Decoded 32 by 32 test pattern from the bundled H.264 video. The inspected block begins at pixel 8,8.

A real decoded frame from the bundled video. We take luma from its 8 × 8 block at (8, 8), subtract 128, then transform and quantize it.

Original luma

Reconstructed

Quantized coefficients · low frequency at top left

-11-2-110008120-10-15160-3060-5-1-50-4-1-11-1-4010003030-102030-2020101060100000-30-10
Step
16
Nonzero coefficients
35 / 64
Toy bit cost
216 bits
Mean squared error
16.59

Orthonormal floating-point DCT, uniform quantizer. Toy code: 64 significance bits; for each nonzero, one sign bit and unsigned Exp-Golomb for |q| − 1. Actual AVC uses specified integer transforms, scaling and context-dependent entropy coding.

Quantization step 16. Move the slider to trade coefficient precision for fewer bits.

Raise the quantization step to turn small coefficients into zeros. The reconstructed block uses the rounded coefficients, and the displayed bit cost follows the explicitly defined toy code.

Quantization divides each coefficient by a step and rounds. Increasing that step reduces precision; many small coefficients become zero. The decoder multiplies the integers by the step, applies the inverse transform and adds the prediction. It cannot recover the discarded remainder. The simplified widget transforms an image block directly, as intra-image coding might; an inter-predicted block would usually transform a residual instead.

Our bit-cost counter has a declared code: one significance bit per coefficient, then a sign and an Exp-Golomb-coded magnitude for each nonzero. This makes each displayed total reproducible. A real codec’s total also includes prediction modes, motion data, partition structure, headers and contextual entropy coding. Sixty-four zero coefficients can be very cheap in a real syntax; they need not cost sixty-four independent bits.

Compress the decisions too

AVC’s CABAC combines binarization, context modeling and arithmetic coding. Common symbols receive shorter effective descriptions under a probability model updated as decoding proceeds. The coded bits are not a list of independent Huffman labels for each DCT coefficient. The specification fixes decoding and probability updates tightly enough that every conforming decoder reconstructs the same values. See the H.264 specification, clauses 8 and 9 for reconstruction and entropy decoding.

AV1, HEVC and VVC improve the available prediction, partition and transform tools in different ways. Their decoder and hardware support matrices differ too. HEVC is H.265; VVC is H.266. AV1 has a public bitstream and decoding specification, §7. Choosing a codec is a deployment decision involving quality, encoding cost, device support and licensing, rather than a bitrate ranking that holds for every clip and setting.

Key pictures are often expensive because temporal prediction cannot borrow the already reconstructed background in the same way. That is why frequent random access points can increase the rate needed for a given quality. It also closes the loop to the ladder: lowering a rendition’s rate changes the encoder’s distortion budget, not just the name of the file. On a phone, the 480p tunnel choice spends fewer samples and often coarser detail to keep the sentence moving.

A codec can be both more efficient and harder to decode. A smaller compressed object saves network work, but a software decoder may need more arithmetic or memory traffic. The best end-to-end choice depends on which fixed-function decoder the phone actually has. The final layer decides whether those compressed bits become a battery-friendly surface or a busy CPU.

Prediction removes redundancy. Quantization discards precision. Entropy coding spends fewer bits describing the decisions that remain.

Level 09/09Silicon

Move pictures without waking every core

At the bottom of the stack, the player’s four-second choice becomes repeated memory accesses and arithmetic on pictures. Software can perform the entire decode, but phones often have dedicated video decode blocks for selected codecs, profiles, levels, bit depths and resolutions. The operating system’s media API routes supported work to that hardware. A codec name alone does not prove that a particular configuration uses it.

These blocks implement recurring operations efficiently: entropy decoding, motion compensation, inverse transforms and reconstruction filters. Implementations vary in which stages are fixed function and which use programmable processors. The useful distinction is that the phone does not need to execute every sample operation as ordinary application CPU instructions. Dedicated paths can reduce energy and free CPU time, but exact savings require measurements on the actual device and stream.

The picture is larger than the packet

A 1920 × 1080 8-bit 4:2:0 picture has one luma sample per pixel and two chroma planes at half width and half height. The nominal sample payload is 1920 × 1080 × 1.5 = 3,110,400 bytes, about 2.97 MiB. Strides, padding, tiling and reference surfaces can add more. At 30 pictures per second, writing even this nominal output once moves about 93 MB/s. Reference reads and extra copies increase traffic again.

That explains why “just copy the decoded frame into JavaScript” is not a free operation. A zero-copy path can keep decoded surfaces in platform graphics memory, passing handles and synchronization fences between the decoder and compositor. The display path samples the existing surface instead of making avoidable CPU copies. Zero-copy names a particular avoided transfer, not a guarantee that no memory movement happens anywhere.

Scheduling stays on general-purpose processors while picture reconstruction and presentation can use specialized paths. The decoded surfaces connect them.

The compositor may sample a Y′CbCr surface, scale it, convert color and blend it with the page. If the platform, transform, color handling, protection and other constraints permit, a hardware overlay plane can scan out the video separately from the primary page surface. It still needs bandwidth and synchronization; it may simply avoid another composition pass. The Linux DRM/KMS plane abstraction documents this distinction between display planes and framebuffer content.

Color is another part of correctness. Bit depth, transfer characteristics, matrix coefficients and range determine how sample values become light. A 10-bit HDR stream cannot be displayed correctly by treating its numbers as ordinary full-range 8-bit RGB. A valid codec and smooth frame cadence can coexist with washed-out or clipped output if the metadata or conversion path is wrong.

The other end of the battery budget

Keep following the bytes outward and the CDN reappears. Our phone shares the talk with many viewers. If their requests use the same immutable versioned segment URL and compatible cache key, an edge can reuse the response. A cache miss may be collapsed with matching in-flight misses so that one upstream request serves multiple waiters. This behavior is an implementation policy; the HTTP caching specification RFC 9111 §4 discusses collapsed requests and their constraints.

Segment reuse scales distribution because viewers can share bytes while making independent timing and quality decisions. Request collapsing can also reduce a simultaneous miss burst.

A VOD segment with a versioned immutable URL can have a long explicit shared-cache lifetime. A live playlist changes as media arrives, so its freshness policy must allow the required update cadence. Signed query parameters, personalized responses and alternate cache keys can fragment reuse. A CDN configuration that treats every viewer’s token as distinct may turn a popular talk into many independent origin misses.

The edge is not the same as the phone’s buffer. A fresh cached segment can be fetched immediately, but only if the phone requests it in time and the radio delivers it. The radio is not the same as the decoder either. A 5 Mbit/s stream can arrive perfectly while a software decode path misses its frame deadlines. Playback telemetry should separate request delay, starvation, decode drops and presentation timing before choosing a remedy.

Return to the tunnel at 5:00. The manifest has already described a 1.2 Mbit/s 480p alternative. An aligned access point lets the player change requests. Buffered media covers the observation and download delay. The container supplies timing and sample locations. The codec reconstructs the reduced-detail picture. A supported hardware path turns it into a display surface. No single layer guarantees uninterrupted video; their contracts give the player a chance to meet the next deadline.

That is why a softer picture can be a successful outcome. The speaker’s next sentence matters more than preserving every edge while the train crosses bad coverage. Adaptive streaming makes that trade deliberately, one finite piece of time at a time.

Check yourselfThe network buffer stays full, but frames drop and the CPU is busy. Which first investigation fits the evidence?

End of Video streaming · 9 levels, Idea to Silicon

Where to next

Browse all 34 divesAll topics