Video and audio formats: containers, codecs and decoding

5 min read

When a device says it can’t play a video, it’s rarely because the file is broken: usually it just can’t read one of its parts. A video has two layers: the container, which is the box, and the codecs, which are how the picture and sound inside it are compressed. Understanding that difference will save you a lot of trial and error.

The container: the box

The file extension (.mp4, .mkv, .ts) tells you the container. The container bundles into a single file the video track, one or more audio tracks, the subtitles, the chapters and details such as each track’s language. It doesn’t determine quality: an MKV and an MP4 with the same content look the same.

MP4 (and its M4V variant): the most widespread. Almost any phone, browser and TV can open it.
MKV (Matroska): open and very flexible. It supports many audio tracks and subtitles of almost any type, which is why it’s common for personal video archives. Some TVs and Chromecast can’t open it as is.
TS (MPEG-TS): designed for broadcasting. It copes well with dropouts because it can be read from any point. It’s the one used by digital TV and by many HLS segments.
Others: WebM (from the web, with VP9 or AV1), MOV (from many cameras and phones), and the older AVI and FLV.

Video codecs

A codec compresses the picture so it takes up a small fraction of the space it would need uncompressed. The more modern it is, the more quality you get per megabit, but also the more work it takes to decompress.

H.264 (AVC): the universal standard for years. Practically every device decodes it in hardware. If you want a video to play everywhere, H.264 in MP4 is the safe bet.
HEVC (H.265): achieves similar quality to H.264 at a much smaller size, and is common for 4K and HDR. Recent devices decode it in hardware; many older ones don’t.
VP9: Google’s open codec, widely used on the web, with efficiency similar to HEVC.
AV1: the newest and most efficient of the four, open and royalty-free. Only fairly recent chips decode it in hardware.
Older codecs: MPEG-2 (used by DVDs and digital TV) and MPEG-4 Part 2 (Xvid, DivX). They still turn up in old files.

One detail that often causes trouble is color depth: 10-bit HEVC is more demanding than 8-bit, and some devices open one but not the other.

Audio codecs

AAC: the default audio for MP4 and for almost all streaming. Compatible with everything.
AC-3 (Dolby Digital) and E-AC-3 (Dolby Digital Plus): multichannel sound, 5.1 and up, from discs, TV broadcasts and streaming platforms. Not every phone decodes them.
DTS and TrueHD: high-quality audio from Blu-ray discs. Few phones can open them on their own.
Opus: modern, open and very efficient, common in WebM. You’ll also see MP3, FLAC (lossless) and Vorbis.

Hardware or software decoding

Decoding means decompressing the video to show it. A phone can do it in two ways:

In hardware: with a circuit in the chip dedicated solely to that. It uses little battery, barely heats up and can handle high resolutions, but it only knows the codecs and profiles it was designed for. When it fails, you usually see a green, black or broken-up picture.
In software: with the main processor. It supports more formats and is more predictable, but it uses more battery, heats up and can struggle with high resolutions.

How Whatplay does it

Whatplay uses the system’s decoders and also includes its own FFmpeg engine for anything the phone can’t open. You choose it in Settings › Player › “Playback engine” and, while you’re watching, in “Options” › “Engine”:

Automatic (recommended): the system decoders first; FFmpeg only steps in for what the system can’t open, such as DTS, TrueHD, MP2, AC-3 where it’s missing, or 10-bit video.
ExoPlayer: only the system’s decoders. Useful for ruling out whether a problem comes from FFmpeg.
FFmpeg: FFmpeg first, for video too, in software. For getting around a phone decoder that’s causing problems.

In Settings › Player you also have “Hardware decoding” (“Turn it off if you see a green or black picture”) and, in “Options”, the “Decoder” selector, with Hardware or Software. If the device’s decoder fails partway through, Whatplay tells you (“The device’s decoder failed: trying the software one”) and carries on without you having to do anything.

To see what you’re watching, open “Options” in the player: under the title you’ll find the resolution, the codec, the Mb/s and the frames per second, when the video provides them. Each audio track shows its codec and channels, for example “AC3 5.1”.

And on the TV

When you cast to a Chromecast, the TV does the decoding, not the phone. Anything the TV can’t open as is (an MKV, an Xvid, HEVC on an older Chromecast), your phone converts while you watch, if Settings › TV › “Convert video for the TV” is on. It uses more battery and warms up the phone. For now, HDR and interlaced video aren’t converted: if the TV can’t open them, they only play on the phone. There are more details in The TV won’t open a video.

What to do if there’s no picture or no sound

Leave the engine on Automatic.
If there’s sound but no picture, or the picture is green or black, switch the “Decoder” to Software.
If there’s a picture but no sound, try the FFmpeg engine.
If nothing works, try the same video on another device: if it doesn’t play there either, the problem is in the file or the stream.

If you’re going to prepare your own videos, H.264 with AAC audio in MP4 is the combination with the fewest surprises. The help sums up the formats in Which formats does it play?, and the technical terms are in the video glossary.