Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

Summary · 6 min read instead of 1 h 4 m

Amin Vahdat is Google's Chief Technologist for AI Infrastructure and, as the episode says, was named head of Google's AI infra at the end of last year. He oversees the TPU program, data center design, networking and power provisioning during a buildout on which Google is expected to spend more than $200 billion in CapEx this year.

Vahdat says Google judges its infrastructure by goodput, the useful work a workload delivers through real failures, rather than theoretical chip FLOPS, because at 100,000-accelerator scale something fails multiple times a day or even an hour. He explains the calculus behind splitting this year's TPU into 8i for inference and 8t for training, and describes DeepMind and Google engineers co-designing in the same rooms and changing chips before tape-out. He argues long-horizon agents are pushing up demand for CPUs, networking and storage alongside accelerators, and names power as the single most fundamental constraint, which Google prefers to solve by working with utilities. He closes with Google's pursuit of orbital data centers and a guess that 2036 racks could be tightly integrated and draw multiple megawatts.

Minute by minute

1. 1:49

   AI data centers are purpose-built. Vahdat says AI and conventional data centers share concrete, electrical and cooling yards, networking and storage, but differ in specialization. A traditional building is a 25 to 30-year investment designed to stay fungible, while an AI building is often co-designed with its hardware. He contrasts storage racks at 10 to 40 kilowatts with TPU or GPU racks already at hundreds of kilowatts and possibly megawatts within a few years.

2. 5:37

   Goodput, not FLOPS. Vahdat calls FLOPS and similar chip metrics theoretical and says Google holds itself to delivered goodput for real workloads under real failure conditions. In synchronous jobs across thousands of components, one failure can halt everything, forcing checkpoint restores and redone work that counts as throughput but not goodput. He says goodput is a Google term that others in the industry are starting to adopt.

3. 10:22

   Failures are a long tail. At 100,000 accelerators, Vahdat says something fails multiple times a day, perhaps multiple times an hour. There is no common cause, he says, but "a long tail of constant discovery": network issues, hardware, and often software such as compiler, runtime, model or operating system bugs. He notes NVIDIA offers a strong reference stack and Google one for TPUs, and most customers use them while many also specialize.

4. 12:52

   Token capacity must double every six months. Vahdat clarifies that Google must double its effective token generation capability every six months, not necessarily its FLOPS. He says as much or more of that comes from software as from hardware, through dozens or hundreds of optimizations. Most gains in intelligence per watt come from the model side, he says, while hardware delivering 2x or more year-over-year acts as a multiplier for everything above it.

5. 15:23

   TPUs began as a contrarian bet. Vahdat recalls that in 2013 conventional wisdom held that you do not build a custom accelerator for a single workload, given Moore's Law and standard programming models. The first TPU targeted inference for language translation and voice recognition, the second added training, and the invention of transformers around then shifted the program. Recommender systems and ads followed, then ever larger Gen AI systems.

6. 18:02

   Why the TPU split into 8i and 8t. About two years ago Google weighed one chip or two for 2026, and chose 8i for inference and 8t for training. Vahdat says the split made sense once serving looked like 30 to 60% of the market over the chip's life; at 2% or 5%, even a 2x faster specialized chip might not justify itself. He adds that each chip can still run the other's workload, preserving flexibility over a six-year hardware lifetime.

7. 21:29

   Transformers rest on linear algebra primitives. Asked why not burn the transformer into silicon, Vahdat says transformers reduce to vector and matrix operations and softmax, which are already roughly baked into hardware. Specializing further to a specific model's layers and matrix shapes is a direction some companies are exploring. He says GPUs are more general purpose than TPUs, Google sells and uses many GPUs, and customers choose by workload.

8. 23:25

   Co-design trades portability for performance. Vahdat says abstraction layers that run on any hardware, software or network bring portability and no lock-in, but leave significant performance on the table. Each layer of the stack may hide 10%, 20% or 2x of gains, which multiply into a large end-to-end opportunity in goodput per watt. He declines to speculate on whether compute stacks explain architectural differences between OpenAI and Anthropic.

9. 26:01

   DeepMind intercepts chips mid-flight. Vahdat describes DeepMind researchers proposing model changes that would benefit from hardware support, leading to joint sessions of days or weeks and sometimes delaying tape-out by a week or two. He says chips sit in a five or six-stage pipeline from concept to production, and working inside one company makes mid-flight changes far easier. He talks to Koray or Demis multiple times a week, and Google uses Gemini to design hardware for future Geminis.

10. 32:45

    TPU primitives have barely changed since v1. Asked how anyone can predict workloads five years out, Vahdat says the TPU architecture at a medium level of detail has not really changed since TPU v1, much like a CPU instruction set. He lists large matrix multiply units, a sparse core for vector and scatter-gather operations, specialized numerics and remote load-store over the ICI network. Researchers know stopping a tape-out is a big deal, so they bring only large gains.

11. 34:01

    Long-horizon agents raise CPU and storage demand. Vahdat says agents remove the human who rate-limits requests, shrinking interaction time from seconds to perhaps milliseconds. The parsing, reasoning and context gathering between model calls run largely on CPUs drawing on DRAM, SSD and HDD, so demand for CPUs, networking and storage is rising alongside accelerators. That forces a choice between mixed buildings and separate buildings linked by costly inter-building networking.

12. 37:43

    Optical circuit switches reroute light. Vahdat says Google brought wave division multiplexing and optical circuit switching into its data centers 15 or 16 years ago. MEMS mirrors redirect light between ports without converting to electrical signals, first to create shortcuts between clusters and to resize the network without moving fiber. For TPUs, a failed rack can be swapped for a spare by redirecting light in milliseconds.

13. 42:29

    Power is the most fundamental constraint. Vahdat says all constraints are hard, but power is the most fundamental. Google prefers grid connection, plans with utilities years ahead and pays for transmission upgrades so others' rates do not rise. If it needs a gigawatt in 2028 and gets 700 megawatts, it may wait or generate some power locally. He says 99.99% reliability on its own would mean building two gigawatts, so sharing the grid benefits everyone.

14. 49:14

    Serving clusters differ from training clusters. Vahdat says older training clusters cannot cover inference demand, so Google builds serving capacity worldwide; serving needs mixed compute, networking and storage and is not necessarily cheaper. He says "our seven and eight-year-old TPUs are still at 100% utilization", though after about six years of depreciation Google replaces whole pods, such as an 8t pod of 9,600 chips across roughly 152 racks.

15. 53:47

    Open standards beat walled gardens. Vahdat says Google avoids forcing lock-in: it likes its JAX framework but supports PyTorch on TPUs through Torch TPU. He cites IP winning in the 1970s and 1980s as an open, narrow-waist standard. He adds that Google's hardware engineers now use AI as much as its software engineers, shortening time from design kickoff to tape-out, and that AI streamlines data center planning.

16. 58:17

    Orbital compute and the 2036 rack. Vahdat says Google is pursuing orbital data centers as a moonshot: space offers about 1.4x solar power and 90% to 100% sun coverage in sun-synchronous orbit versus roughly 28% to 35% on land. Cooling, repairs and networking get harder, but he sees no fundamental showstoppers. For 2036 he imagines centrally manufactured, tightly integrated racks, possibly multiple megawatts each, plugged into water, power and fiber.

Published Oct 6, 2026.