FFmpeg: The Incredible Technology Behind Video on the Internet | Lex Fridman Podcast #496

Summary · 5 min read instead of 4 h 18 m

Kempf is lead developer of VLC and president of VideoLAN. He joined the project in 2003, moved it into a non-profit, and now works on the open source low-latency streaming project Kyber. Kunhya is a longtime FFmpeg contributor and codec engineer, runs the FFmpeg account on X, and started a company building broadcast equipment.

Kempf and Kunhya explain how video playback works and why FFmpeg and VLC sit under most video on the internet. Kempf tells how VLC grew out of a student project at a French engineering school and why he refused offers to add ads or bundled software. Kunhya describes the dispute over Google's AI-found security reports. Both argue that handwritten assembly, as in the dav1d AV1 decoder, is still needed for speed. They also cover reverse engineering of codecs, the AV2 and patent landscape, Kempf's low-latency work at Kyber, and video archiving.

Minute by minute

1. 5:24

   VLC plays almost any file, even absurd ones. Kempf says VLC handles VHS capture cards, DVD audio and obscure game codecs. At a VideoLAN conference the team ran a contest for the worst possible files, including an MKV that changed resolution and rotation on every frame, and VLC opened them all. He says about 25% of traffic to the VideoLAN site comes from people searching for the cone player.

2. 11:33

   Playback is demuxing, decoding and heavy compression. Kunhya walks through the stages: fetch the byte stream, demux the container, probe whether the GPU can decode the codec, then fall back to software decoding. He says up to 45% of files are not GPU decodable. Kempf says video is compressed 100 to 200 times and each codec generation saves about 30% for the same quality, at far higher CPU cost to encode.

3. 30:02

   FFmpeg is the shared library under most video. Kempf describes FFmpeg as low-level codec, mux and filter libraries plus command-line tools, used in VLC, Chrome and many services. Both stress that open source licensing is a social contract. Kempf moved libVLC from GPL to LGPL to let companies embed it, which meant tracking down more than 350 contributors. Kunhya says FFmpeg cannot relicense because it has thousands of contributors.

4. 55:35

   Kempf refused offers that would have ruined VLC. Kempf says he turned down tens of millions of dollars, because the offers came from ad companies wanting toolbars, search changes or ads in VLC. He tells the history behind it: a student streaming project at École Centrale Paris in the 1990s became VideoLAN, which became VLC. He says he took it on in 2003 and made it a non-profit, and that selling out would have betrayed the other contributors.

5. 1:10:03

   Google's AI security reports strained FFmpeg volunteers. Kunhya says Google used AI to file security reports against FFmpeg, set a 90-day deadline, and publicised its AI before fixes landed, over a bug in an obscure 1990s game codec. He argues that companies with the resources should send patches or money. Kempf says Google has since started sending patches and paying rewards for fixes, and that the public pressure raised donations and awareness.

6. 1:35:51

   VLC and FFmpeg depend on each other. Kunhya calls them a binary star system and says VLC is to FFmpeg as Android is to Linux. VLC gives FFmpeg reach and odd test files, and has used donation money to fund its development. Kunhya estimates over 80% of FFmpeg pipelines depend on x264, a VideoLAN project. Kempf says compiling VLC for Windows pulls in about 16 million lines of code.

7. 1:43:43

   Reverse engineering codecs takes patient guesswork. Kempf tells how Kostya reverse engineered the GoToMeeting codec in about two months after Kempf put up a bounty. Kunhya describes the method: hook the vendor's decoder to dump raw YUV, then compare output bit for bit while stepping through binaries of millions of instructions. He also explains bit exactness, where every decoder must produce identical output. FFmpeg's FATE tests run on volunteers' machines.

8. 2:04:15

   Handwritten assembly still beats compilers. Kunhya says SIMD assembly gives speedups of 10x to 50x over C, and shows one function at 62x. Kempf says dav1d has about 30,000 lines of C and 240,000 lines of handwritten assembly, and uses its own calling convention. Kempf says auto-vectorization is not close, and that with hardware speedups slowing, optimizing for the CPU matters more.

9. 2:25:14

   Rust suits new code, not rewrites. Kunhya compares Rust to the Sinclair C5 and says it must match existing tools in tests, ABI stability and compilers before projects move. Kempf has written Rust modules for VLC and builds Kyber mostly in Rust. He calls it good for new projects, poor at interoperating with existing code, and says rewrites underestimate hidden logic. He notes that handwritten assembly breaks Rust's safety model.

10. 2:50:45

    x264 won by tuning for how video looks. Kunhya says x264 broke with industry habit of optimizing mathematical metrics like PSNR, which produced blurring. It used psychovisual rate distortion and adaptive quantization, tuned by hobbyists on laptops with samples like Park Joy. Both say anime fans were a large part of the community. Kempf says each new codec generation, such as AV1, gives roughly 25% to 50% better compression.

11. 3:10:57

    A fake VLC was used for targeted attacks. Kempf says WikiLeaks documents showed the CIA used a modified VLC with one extra DLL that read a user's documents folder and sent them out. Chinese hackers used a similar approach against Indian targets, which got VLC banned in India until Kempf fought it in court. Kempf adds that VLC is being split into separately sandboxed processes for decoding, demuxing and filters.

12. 3:26:58

    Kyber targets four-millisecond remote control video. Kempf says Kyber is open source and aims at remote control of robots, drones and cloud games. It sends video, audio and control inputs over one QUIC connection, with clock-drift handling and forward error correction. He has reached 7 milliseconds glass to glass, with roughly 3.5 in the NVIDIA encoder and 2 in the Intel decoder, and targets 4. It is dual licensed under AGPL and commercial terms.

13. 3:39:02

    AV2 aims for 30% over AV1. Kempf says AV2 should cut bandwidth by about 30% against AV1, and H.266 by about 30% against H.265. A dav1d successor is planned and VLC showed an AV2 demo at CES. He says MPEG patents spread across several pools, including HEVC Advanced, pushed Netflix and YouTube toward the royalty-free AV1. VLC avoids the issue partly because France rejects software patents.

14. 3:59:02

    Archivists rely on FFmpeg and lossless FFV1. Kunhya says the archiving community, led by Dave Rice, values open source because it lacks budgets and fears lossy compression changing the record. It funded the FFV1 lossless codec and GPU encoding for it, and shares workflows with countries that cannot afford archive institutions. He cites the BBC Domesday project, which took about 20 years to become readable again, as a warning.

Published May 6, 2026.