Real Alpha

Training Data · 4 min instead of 1 h 4 m · Oct 6, 2026

Google split the TPU into 8i and 8t based on market size

AV

Amin Vahdat

Chief Technologist for AI Infrastructure, Google

  • Sonya HuangHost · Partner, Sequoia Capital

    Works with OpenAI and Anthropic; co-wrote Sequoia's essay on generative AI.

About this segment

Vahdat walks through the calculus of whether to specialize a chip: how large and durable the workload is, and how much faster the specialized chip would be. Inference grew enough that two chips made sense.

From “Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI”, Oct 6, 2026. Watch the full episode

“if we projected inference to be 2% or 5% of the market, even if that specialized chip is, let's say, 2x faster, it might not make sense, right?”
AVAmin Vahdatat 19:16 · Training Data19:16

Takeaways

  1. Specialize only if the workload is big and durable, not a three-month spike
  2. Inference was projected at 30 to 60% of the market, so a 2x chip pays off
  3. Each chip can still do the other's job, which limits forecasting risk
Watch 3:33 from 18:12

In this episode

Full episode
  1. 5:27Goodput, not FLOPS, is how Google holds itself accountable5:03

  2. 18:12Google split the TPU into 8i and 8t based on market size3:33

  3. 34:14Long-horizon agents push CPU, storage and network demand up3:48Up next

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 18:17

    I mean, I think that of the chip to a particular workload. So in other words, in the end, we've been thinking about this for every generation. Would we further specialize? The question that we were facing, let's say, two years ago or a little bit more was, in 2026, should we have two chips or one? We could have one chip that could do inference and training simultaneously and do both quite well. Or we could have two chips, one that was further specialized for inference and the second one that was further specialized for training. This analysis and this work led to the release of two chips this year, 8i for inference, 8t for training. And in the end, what we realized is that it wasn't the case necessarily a couple of years ago, but by 26, we saw inference and serving really taking off. And so having a chip that would be significantly faster for serving that we

  2. 19:13

    thought might be 30, 40, 50, 60% of the market in its lifetime, started making a lot of sense for us relative to if inference, if we projected inference to be 2% or 5% of the market, even if that specialized chip is, let's say, 2x faster, it might not make sense, right? Because you'd actually go for the one general purpose chip that's not fully optimized, but that's okay because you now have the uniformity, et cetera. So really, it does come down to a calculus question of, okay, would you further specialize? How big is that workload? How big is that workload projected to be in a couple of three, four years? Is this a sustained growth for a particular workload?

  3. 19:58

    In hardware, again, as you know, there is this opportunity where the more you specialize to a particular workload, the less flexible it is, the faster, the more power efficient the hardware is going to be. So it is this art, and it's this projection of what are you designing to, and how persistent is that workload. In other words, if it's going to go away after a month, or two months, or three months, even if it's big for those three months, you got a really narrow window to intercept it. So it has to be somewhat durable, and you have to be able to project exactly what win you can get for specializing to it.

  4. 20:34

    So the rough trade-off is there's a large fixed cost to support a new program. Yes. And you basically have to think that there's going to be enough demand for that specific program to justify the cost. Exactly. And to some extent, for example, for our 8i and 8t chips, both chips can do the other workload. This is also key. If 8i could only do inference and it couldn't do training at all, in other words, it had zero performance for training. And then vice versa, if AT was amazing at training, it had zero performance for inference, that would have also been a key limitation. Why?

  5. 21:08

    Because we would have to predict ahead of time, over a six-year period, lifetime of the hardware, exactly how much we need of each. In this case, it was nice because both chips are better at what they're specialized for. But if you needed to, if you had leftover capacity in one place or the other, both can actually do the other's job. Depending on the specialization you do, you might get so specialized that you actually limit yourself from being flexible, fungible.

  6. 21:35

    And if the rough trade-off then is kind of size of program on the other end, like it seems to me that so much of the modern AI market is transformer-based.