Why hand-written assembly beats compilers by 10x to 50x

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 2:04:04

   fighting like the fact that you need to buy something new non-stop while you could optimize more which is a lost art you gotta tell me about this lost art or this uh the the carriers of the flame of assembly what what is what is assembly why is it beautiful why is it challenging How does it work? So when you write assembly code, you write this using the instructions the actual processor is using directly.

2. 2:04:36

   So most of the time you would write in a language. Let's take C as a good example. The compiler would use that to create assembly language and machine code instructions for you based off your C code. And there's a specific flavor of assembly that we use in FFmpeg. That's called SIMD, single instruction, multiple data. so this means for example say say i want to add five to a number in scalar assembly so this is what's known as you work on an individual element so i want to have a number of i have a number 10 and i want to add five i use the add instruction and i add five to 10 and i get 15 with simd i can have a whole vector of 16 different numbers that could all be different if i want to add five to that i can i can run one instruction and that one instruction sums all 16 elements and that

3. 2:05:30

   as you can imagine lends itself very well to video video video is you know a pixel grid so i can perform operations on multiple pixels at the same time the key thing that we do differently in ffmpeg is we don't use any abstractions or any major abstractions on top of that so there's a part of the world that uses what's known as intrinsics so these are c functions that behave very similarly but not quite the same to writing assembly by hand so the registers that data is stored in on the cpu the compiler allocates those for you and so the key thing to understand was when we write simd is we have a 10x and not percentage 10x to 50x speed improvement that that function is 62x that's nuts on the ffmpeg account as you know posts and tweets a lot about that to try and say hey we're doing this stuff you are a person who sees the beauty in assembly but it's also

4. 2:06:28

   extremely useful for these kinds of applications to actually significantly outperform even c which is crazy it is necessary yes right because like one of the projects um that we need to talk about is called David. So David is a decoder for the format that was done by Allianz for Open Media, which is a video decoder called AV1. So for people who don't know, we've been talking about H264. AV1 is another hugely popular standard and codec that is increasingly taking over the Internet.

5. 2:07:08

   And when this format was launched, many people said, especially even from the Alliance for Open Media, right, which is Google, Netflix, Amazon, Mozilla, they say, well, this format is so complex, it must be done in hardware to do decoding, right? And, well, I arrived with a few other people, mostly Ronald, Henrik, and Martin, and we said, we need to have an extremely good software decoder because it's going to take time to have hardware.

6. 2:07:37

   And so we wrote this project, which is beyond insane. We are talking about 30,000 lines of sea, but 240,000 lines of handwritten assembly, right? Handwritten assembly, 240,000 lines. That's incredible. I mean, some of the stuff we're talking about is probably the biggest assembly code bases. To give you an idea, and Kieran can correct me, but I think the FFmpeg has 100,000 lines of assembly for all the codecs.

7. 2:08:13

   All codecs. And just this one has 240,000. It's a video land project, of course. And it is optimized at the maximum because the motto when we're starting the project is every cycle matters, right? Every cycle matters because David is used in VLC and in some software AV1 playback stacks. We are talking about probably 3 billion devices which are going to decode video nonstop because, for example, 30% of the video from Netflix are now in AV1, 50% of YouTube, right?

8. 2:08:49

   And you often don't have a hardware decoder because not many devices have a hardware decoder. And with David, we realized that in one or two cores, you were able to decode 720p correctly. So it is like literally incredible, right? That's David. Look at that. Yeah. So this is another spicy tweet from you. This is what peak video codecs should look like. 79.9% assembly, 90.6% C, and 0.5% other. And what's incredible is with those tweets, which is factual, people get crazy.

9. 2:09:28

   They are unhappy, right? For the last two years, they go crazy. No, intrinsics is fine. The compiler is fine. Oh, they go. You can optimize your compiler. Auto-vectorization is your fault. You don't understand. And we've tried that forever, right? For two years and two years later, showing hundreds of examples of handwritten assembly. No, no, no. You're doing it wrong. The compiler can do this. so we should actually just articulate a little clearer so the intuition there from the software engineering folks when you have code like okay let's just take an example c plus plus there's a compiler that's doing a lot of the optimization yes and the presumption is if you have a good enough compiler if you continue to improve the compiler you're going to generate code yeah they can perform like optimal performance you cannot possibly beat it and you're consistently challenging

10. 2:10:16

    that thought that if you do by orders of magnitude by orders of magnitude handcrafted assembly can outperform c the two things that they tell us is yeah but modern compilers have auto vectorization right because simd that we're doing is vectorization and like it's not even close right it's not even close right it's not like five percent ten percent slower it's multiple times slower so can we i don't know if you can say something philosophically because there's a lot of But...

11. 2:10:43

    There's a lot of great software engineers, great engineers, great machine learning people. Karpathy will listen to this and say, what's the intuition he's supposed to get from this? What are we supposed to... Karpathy learned assembly because of the tweets, by the way. He's like, oh, I think this is a move. Let me figure out what's happening here. And you know the way he documents his work and so...

12. 2:11:02

    Philosophically, what's important to realize is that we passed the time where hardware was going so much faster, right? We are at the end of the more low. we have limitation for for ai for memory you need to go down in the stack and optimize more to get more power from what you have because our requests for power cpu power gpu power are exploding while the hardware is not exploding in speed right so you what people do is that they add more calls right but that's basically like at some point you can have 250 calls right so what we do is to take every inch of the machine not just that not just that we abuse the machine we go and use we use the machine in ways that the the creator didn't expect sometimes we use an instruction that's completely unrelated to what we do we use a cryptography instruction in video processing

13. 2:11:56

    to do nothing related and one of the other things that we do for example in david which is a bit crazy is that we don't use the function calling convention from the operating system we should explain that that is extremely complex but basically usually when you do move from one function in code to another there is a way to save the registry the state of the cpu to enter another function um and this is like standard it's a bit complex i would simplify this a bit so so david does things to abuse the calling convention you could define the calling convention as i've written a function and i want to call another function how is the data shared between the functions because there's a convention what's known as a calling convention and what david does for optimal reasons is create its own calling convention sometimes so if i want to call

14. 2:12:49

    lex friedman's library we got we've got to agree on a convention so that i can share data with you in the assembly language space and one of the challenges in assembly is every operating well not every operating system but there are at least four that i can think of on x86 linux 32-bit windows 32-bit windows 64 linux 64 they all have their own calling conventions and so one of the amazing things lauren merritt did who we talked about before was create a very lightweight abstraction layer so you could write your assembly code once and it handled all the calling convention stuff for you which was always a problem because you had to manage four different variants but david takes this even further for speed reasons it does its own calling convention um within itself to bypass the kind of rules the rules of the rules of sort of functions and say

15. 2:13:37

    okay actually i'm going to call a function this way because i know it's within my library does it have to be special to every single operating system well if it's custom no but the the challenge is in general yes and in terms of in terms of each instruction set so the thing to also emphasize is we do this on every instruction set so every instruction set has its own handwritten assembly which is even more crazy and that that that has um that matrix has got bigger in recent years because of risk 5 because of arm 64 because of the new sve there's sme x86 has avx 512 avx so we do runtime processor detection we see what the machine ffmpeg is running on or david's running on is capable of because you could be on a laptop from 2008 where this isn't there runtime detection we set function pointers accordingly and then from then on off you go or you could be on a machine

16. 2:14:31

    with risk five yes and in all that we don't even respect the calling convention of the operating system in order to be faster because we know that we are going to be called from within our binary