Frontier training costs rise 4x per generation, and AI speeds the work

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:57

   well i think that it depends if the frontier keeps moving so each year it's actually more capital so um and i'll get back to the question on the resourcing between let's say pre-training and post-training i would say that a year ago or maybe 18 months ago it would have been and i say orders meaning that order of 100 million, so hundreds of millions of dollars. I think that now, or let's say even six months ago, is probably order billion, so billions of dollars, single-digit billions.

2. 9:27

   Going into next year, I think order 10. You can kind of think about it as for every generation of model, there's a 4X multiplier in compute, roughly. And one heuristic that I have is like when, But let's say we talk about the chips that are currently considered, the ones that a frontier chip, and maybe a couple of years ago it was an H100, and maybe it was 100,000 H100s. Then Astra was trained, and that was just a big run, on 100,000 Blackwells, which is roughly a 4x multiplier on an H100.

3. 10:01

   Now the next generation is going to be around 100,000 Vera Rubens, I believe. So that kind of gives you the scale, and it moves from hundreds of millions to billions to tens of billions. Yeah. It seems like at some point that has to asymptote then because ultimately relative to revenue scale of these companies, even potential revenue scale, you start to tap out in terms of the ability to invest against what you're actually going to produce from an economic value perspective.

4. 10:23

   Absolutely. So do you think we're heading into that asymptote in terms of model size or at least training cluster size? Absolutely. I think that in terms of, you know, there's only so much CapEx that you can do, right? And Frontier Labs are now putting hundreds of billions of dollars into it. So will they be putting trillions of dollars in the next year or two? It's probably unlikely, but there are a couple of things that are happening.

5. 10:49

   One is that the use of that compute is becoming a lot more efficient because the models are themselves becoming tools that help build themselves. So I think that when it was just human researchers doing the work, there's probably a 7x improvement each year. And you can actually track these things very technically. In pre-training, you do it by you have your pre-training loss, and then you measure your compute efficiency gains against you made some changes.

6. 11:19

   Now you have a new pre-training loss, and it should be that you got to the same point with less compute. And so when you say 7x, it's actually a very concrete thing that you're studying. and now I would say across the systems of, you know, I mean, pre-training is probably a bit more hardened, but there's a lot of headroom and reinforcement learning. We're probably at a point where it's, you know, 30x like efficiency, depending how good your model is for actually improving itself.

7. 11:46

   So that is one thing that's happening. 30x per year or over what? Yeah, something like that. Yeah, I mean, it depends on, you know, what metric it is, but it roughly, I do think that the speed is probably four or more times faster than researchers just doing it alone, and that'll probably accelerate. So the amount of intelligence you can extract per training flop is increasing. So at some point, that asymptote is okay. And then also the amount of revenue that each flop can generate also increases, right? The more you pack intelligence. So going back to the question around how does this distribute across pre-training and reinforcement learning? And to train Beam, which is a 500 billion parameter model total, 23B active, it was 6,000 GB300s. We ran it, I think, for a few weeks, but now with infrastructure efficiencies, we can do it in about

8. 12:42

   12 days, maybe less. So it's a coupling of scientific and infrastructure efficiencies. reinforcement learning was a little over 10 000 gb 300s for four weeks so actually there were more