Video transcoding, explained

Transcoding, transmuxing, transrating

Three similar words describe three different amounts of work. Transmuxing (remuxing) changes only the container: an H.264 stream is copied from MKV into MP4 or cut into HLS segments without being decoded. Transrating re-encodes at a different bitrate but the same resolution and codec. Transcoding in the full sense decodes to raw frames and re-encodes, possibly with a different codec, resolution, frame rate or colour format. A streaming ladder needs the full version, because it produces multiple resolutions from one source. The free video converter is a small everyday example. It decodes your file and encodes it again, for instance to H.264 and AAC in an MP4.

The cost follows the same order. Transmuxing is limited by disk speed. Transrating and transcoding are limited by the encoder. As a rule of thumb, encoding H.264 at 1080p with a careful software preset runs at about real time on a modern processor. AV1 is slower still, as the codec comparison discusses.

The pipeline, stage by stage

1. Probe

   Read the container and stream metadata: codecs, resolution, frame rate, whether the frame rate is constant, interlacing, colour primaries and transfer characteristics, audio channels, duration. Everything downstream is configured from this. A corrupt index or a missing duration is caught here.

2. Demux and decode

   Split the container into elementary streams and decode the video to raw frames, typically 8-bit or 10-bit YUV in 4:2:0 chroma. Hardware decoders speed this up considerably; software decoders handle the exotic cases.

3. Filter

   Scale each rendition with a good resampling filter. Deinterlace if the probe found interlaced fields. Convert the frame rate, or force a constant one for variable sources. Tone map if HDR must also be delivered as SDR.

4. Encode

   Compress each rendition with a codec and rate-control settings chosen for streaming. Constrained bitrates keep the BANDWIDTH a playlist advertises honest. A fixed keyframe interval makes every rendition segment at the same instants.

5. Encode audio

   Decode and re-encode the audio to AAC, normally at 128 to 192 kbps stereo, shared across all video renditions. Multiple language tracks become alternate renditions.

6. Package

   Segment each rendition, write the media playlists and the multivariant playlist, generate thumbnails and scrub previews, and upload everything to storage from which it is delivered.

7. Verify

   Probe every output: duration matches the source, segment counts match across renditions, keyframes sit at segment starts, audio and video stay in sync. Then mark the video ready.

Rate control: CRF versus constrained bitrate

A one-off encode for archiving uses constant quality (CRF in x264 and x265). It spends bits wherever the picture needs them and lets file size fall where it may. Streaming cannot use pure CRF, because the playlist must advertise a peak bitrate that the player can rely on. The usual compromise is capped CRF or constrained VBR. That is a quality target plus -maxrate and -bufsize limits, which bound how far the bitrate can spike over any short window. The buffer size models the decoder's buffer. A rule of thumb is one to two times the target bitrate.

Two-pass encoding analyses the whole file before encoding it. It gives a more even result at a chosen average bitrate, but it doubles the time. Some platforms go further with per-title or content-aware encoding. They analyse each source and choose bitrates for its own ladder, instead of one ladder for everything. A static talking head and a sports match do not need the same bits at 720p. The bitrate guide gives working ranges.

Keyframe alignment: the rule that makes switching work

Every segment must begin with a keyframe, called an IDR frame in H.264 terms. Every rendition must also place its keyframes at the same timestamps. If not, a player switching renditions at a segment boundary would land in the middle of a group of pictures. It would have nothing to decode from. In ffmpeg with x264, you fix the GOP length with -g and -keyint_min. You also turn off scene-cut detection with -sc_threshold 0. Then the encoder cannot add an extra keyframe at a scene change in one rendition but not in another. -force_key_frames with an expression is the alternative when the interval must be in seconds rather than frames.

The keyframe interval also sets the minimum segment length. Apple recommends keyframes at least every two seconds with six-second segments, so each segment holds three groups of pictures. Longer intervals compress slightly better. Shorter intervals make seeking and switching snappier. Two seconds is the widely used compromise, and the adaptive bitrate guide explains what the player does with the result.

Settings that decide the output

Where pipelines go wrong

• Variable frame rate sources, common from phones and screen recorders, produce renditions whose segment boundaries drift apart unless the pipeline forces a constant frame rate first.
• Interlaced sources that are not deinterlaced show combing on every progressive display; the AVI conversion guide covers the filter.
• Scene-cut keyframes left enabled in one rung and not another break alignment silently; playback works until the first switch across the misaligned point.
• Upscaling a low-resolution source to fill a ladder adds bytes without detail. Renditions above the source resolution should be skipped, which is why a 720p upload does not gain a 1080p rung.
• Double compression. A source that was already heavily compressed gives the encoder artefacts to keep instead of detail. The blurry after upload guide shows the effect, and the preparation guide the remedy.