Goodput, not FLOPS, is how Google holds itself accountable

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 5:31

   How do you measure and hold yourselves accountable in the middle of this big build out? We were talking right before the show about how flops is a vanity metric and you prefer an alternate metric. Can you say more? Yeah. So I think one thing to note is that, you know, whether it's flops or pick your other favorite chip centric metric, these are in theory. In other words, under some conditions for whatever chip you have, this is the maximum amount of flops that you can deliver.

2. 6:01

   What we really care about in the end is what's the performance delivered by workload? That's what we're looking at. and rarely it happens, but actually relatively rarely, is that performance determined by a single chip. It's flops, it's HBM, it's SRAM capacity, etc. Those are all super important, but it might be how 2, 4, 8, 16, 1,000, 10,000 more of these chips composed together with not just the accelerators, again, whether TPUs or GPUs, but then the CPUs that might feed them the data, the network that connects them all together. So what is the workload that you're running and what is the performance of that workload? One measure that is interesting is what's your flop's utilization?

3. 6:48

   So for example, for a particular workload, if you have in theory a teraflop or a petaflop capable, for your workload, what fraction of that are you delivering? Now that's a measure of good put. What is good put? So you can think of throughput, which is a well-known term. That's throughput that is possible but now let's consider some uh other considerations one is what is the slowdown of the workload again just inherent to the workload but other aspects of it that really hits our reliability so in terms of how we hold ourselves accountable if we have a chip failing for a synchronous workload and many of these workloads whether training or serving or agentic workloads they're synchronous there's many many components that are working together simultaneously Now, if you have, let's say, 1,000, 10,000, 100,000 of these components working together

4. 7:38

   simultaneously, and they're really needing to coordinate at microsecond or millisecond granularity, one of them fails. It might actually bring the whole thing to a stop, because everyone is counting on everyone else to do that part of the job in order to come up with the answer to a really tough question. One of them stops. Okay, now we have to figure out what happened, which one stopped, what's a checkpoint that we have of the computation at some previous state and time? How do we restore that checkpoint?

5. 8:06

   How do we restart? Worst case, we have to restart from the beginning. That would be really bad, but that can happen in certain cases, especially more on the inference side. The point here is that if you now then have to go back and redo a bunch of computation, if you then have to pause and wait to figure out what happened for the failure, all that work is work. It's not actually helping you get the answer, right? In terms of, let's say that you're working out a problem on paper. Step one, step two, step three, step four. If you have to go back to step one because you've made a mistake, yes, you're still doing work. That's throughput.

6. 8:42

   You're doing work, but what is the good put in delivering your answer? It's basically the total amount of time it took you to solve the problem. That's what matters. And if you have failures, if you have failure recovery, if you have whatever it is that's interrupting your work, That's part of the issue. So now, how do we hold ourselves accountable? It's delivered good put, not theoretical benchmark throughput or goodness in theory, but actually for a workload, for real failure conditions, what's happening.

7. 9:12

   And the unfortunate reality is that at 100,000 accelerator scale. I was going to say at that scale, something's failing all the time, right? All the time. And each one of these, as you're aware, each one of these chips is a wonder of nature. I mean, they're at the very bleeding edge of what's possible to manufacture. And so, again, to me, it's stunning. And now, actually, these chips aren't just one chip.

8. 9:36

   They're packages made up of, in many cases, two, four, eight, maybe more chiplets that are composing together. And then, of course, HBM off to the side, network connectivity, maybe things like co-packaged optics. No criticism. Lots of things can fail. And now if you've got 100,000 of them, something is going to fail. You have to be prepared for that, detect that in near real time, recover from that in near real time.

9. 10:02

   Doing all that, it's really like finding the telemetry problem is massive. It's like finding a needle in the haystack continuously across certainly seconds and minutes, but for some of these jobs, hours and days and even weeks, just continuous and online. How we hold ourselves accountable is what's the good put that we deliver for the workloads that actually matter in the data center. Is good put a Google term or is an industry term? It's a Google