Real Alpha

The MAD Podcast · 2 min instead of 1 h 17 m · Oct 8, 2026

Text-to-SQL reached 99.5% at a bank, but at great cost

AP

Andy Pavlo

  • Matt TurckHost · Partner, FirstMark

    Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.

About this segment

Pavlo explains why he distrusts text-to-SQL academic benchmarks. A large bank lifted accuracy from 60% to 99.5% with a semantic layer, but it took years and millions of dollars.

From “Andy Pavlo: What Happens When Billions of AI Agents Hit Your Database”, Oct 8, 2026. Watch the full episode

“they were able to get it up to like 99.5%, which is insane. I think that is, I think that's a viable thing now if you invest the time”
APAndy Pavloat 24:55 · The MAD Podcast24:55

Takeaways

  1. Benchmarks like Spider are in the training data, so results mislead
  2. The bank's 60% to 99.5% came from a rich semantic and context layer
  3. Off-the-shelf tools won't match that without the same investment
Watch 2:11 from 23:38

In this episode

Full episode
  1. 10:43Agents could mean 10 to 100x more database queries4:37

  2. 23:38Text-to-SQL reached 99.5% at a bank, but at great cost2:11

  3. 33:51LLMs tune databases 85% of the way in 15 minutes2:02Up next