Long-horizon agents push CPU, storage and network demand up

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 34:18

   changes in workloads over the last year or so, I think it really started at the beginning of this year, this calendar year, was kind of the rise of the long horizon agent. Yes. And I would guess that's a very different shape of workload than kind of the quick turn LLM conversations of years past. What does that mean in terms of data center needs? Yeah, so I think there's two huge aspects to this. One is it's no longer human to, whatever you want to say, accelerate interaction. In other words, when you are typing at a prompt on a web browser or on your phone or whatever, of course, in response to your prompt, there's going to be a bunch of work that happens.

2. 34:57

   But then when the response comes back, you've got to read it. You've got to think about it. and then maybe you have a follow-up. That's going to be multiple seconds of interaction time. Now in deep horizon agents, there's no human loop that is going to naturally rate limits how quickly requests are going to go to the model. So what went from seconds, maybe tens of seconds in terms of interaction time, is now going into perhaps milliseconds. As soon as I get a response back, I can parse it, I can reason about it perhaps a bit, and then I can figure out what my next request is going to be.

3. 35:31

   So that's big change one. Big change two is all that reasoning and all that parsing is going to probably happen on a CPU. And that CPU is then going to have to probably think about, OK, what other state do I need to go gather before making my next prompt back into the model? Like in other words, I've learned something from this response. I'm going to take another step, but I actually need to go grab some context, maybe from DRAM, local to me, maybe from someone else's DRAM on another CPU, maybe on SSD or maybe on HDD somewhere else.

4. 36:03

   So now a huge amount of orchestration has to take place as well. So the design actually is changing pretty significantly where the demand for accelerated computers going up, but the demand for CPU and networking and storage, the traditional data center CPU, et cetera, is also going through the roof at the same time. You mean you're putting more CPUs alongside your GPU racks then? So this is GPU, TPU racks.

5. 36:30

   This is the key question is going back to this question of optimization and specialization. If we start putting lots of CPU racks, and it is a question next to the GPUs and TPUs, that means that actually we can't fully specialize to the density and network requirements, let's say, of a TPU rack relative to a CPU rack. TP rack is going to be more dense than a CP rack. It's probably going to need more networking than a CP rack.

6. 36:57

   So in other words, now our building design is going to change. Another option is maintain your uniformity. Put your TPUs or your GPUs all in one building. And the building next door, maybe, put your CPUs. And maybe, by the way, the hard drives have to be in another building on the other side because they have yet another set of requirements. Now you need networking between these buildings at a pretty significant level.

7. 37:25

   So once you leave a building, the networking complexity goes up significantly. From a reliability perspective, from a cost perspective, latency goes up probably acceptably, but still now you might go to hundreds of microseconds, potentially more with queuing between the components. So the considerations do change in pretty interesting ways. Your job is hard. That's fun. It's fun. What's happening on the networking side?

8. 37:54

   I've heard that Google's always been at the forefront of the newest in networking, including optical.