Real Alpha
Watch mode

Training Data · 4 min instead of 1 h 8 m · Oct 13, 2026

Simple rewards in complex worlds beat hand-crafted RL objectives

David Silver

Co-founder, Ineffable Intelligence

Led the DeepMind team behind AlphaGo and AlphaZero and ran its reinforcement learning research until 2026.

  • Sonya HuangHost · Partner, Sequoia Capital

    Works with OpenAI and Anthropic; co-wrote Sequoia's essay on generative AI.

About this segment

Silver argues against breadcrumbing agents toward set goals, as LLM RL does, and for simple rewards in rich environments. He uses evolution and the human chain from trade to science as the model.

From “David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience”, Oct 13, 2026. Watch the full episode

“the pursuit and optimization of a very simple, easy to describe quantities can often give rise to incredibly hard to describe complexity and emergence.”
David Silverat 18:15 · Training Data18:15

Takeaways

  1. Most LLM RL pulls the agent by its nose toward a chosen outcome
  2. Pair a simple objective with a complex environment to get emergence
  3. Coordination led to language, trade, arithmetic, tools and science
Watch 3:58 from 16:09

In this episode

Full episode
  1. 27:19Dropping human data let AlphaZero beat two years of work in hours4:45

  2. 16:09Simple rewards in complex worlds beat hand-crafted RL objectives3:58

  3. 47:38Chain-of-thought monitoring will fail as agents grow more capable3:27Up next

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 16:14

    What do you think the agents will rediscover first when that's going to be your first moment of, oh, this is working? I imagine we're going to be surprised. I think one thing which is different about the way we take to doing reinforcement learning to a lot of the rest of the field is that we are not breadcrumbe trailing the particular objective to be learned about. A lot of the way people do RL with LLMs at the moment is to kind of say, oh, you know, the agent should be learning about this problem and then this problem and this problem.

  2. 16:46

    And after kind of following that trail, we're kind of pulling the agent by its nose towards a particular desired outcome. We're not taking that approach. We're giving the agent some simple objectives and asking it to discover things that will surprise us and amaze us. So the agent is, in principle, capable of discovering all kinds of things. And we don't necessarily know in advance what those discoveries will be.

  3. 17:10

    And it's kind of just going to be exciting to witness that journey. So I hope to be surprised, I think, is the answer to your question. Interesting. So what you're saying is instead of explicitly giving the agent the goal to go rediscover language or, you know, build a calculator or build a computer, you're going to give it other more open-ended goals. And the agent may discover that the most efficient way to actually or the best way to actually accomplish that goal, maybe to build some of these tools that we as humans have already built.

  4. 17:40

    Is that one way of putting it? That's one way of putting it. I think another way to say it is people often think in RL, oh my God, how do I come up with a reward function that kind of perfectly specifies this really complicated thing that I want to discover? Like, you know, how can I specify exactly what I mean by science? Or how can I specify exactly what I mean by interesting maths? But there's an alternative.

  5. 18:01

    And the alternative is that to give the agent something, you know, really straightforward to optimize, but in a really complex environment where there's a lot to learn about. And so it turns out that, you know, if you look at the way the world has worked in nature, the pursuit and optimization of a very simple, easy to describe quantities can often give rise to incredibly hard to describe complexity and emergence.

  6. 18:31

    So examples of that would be, you know, evolution, which kind of just optimize a kind of survival of the fittest, you know, some kind of fitness signal. And the pursuit of fitness gave rise to all of the complexity that we see in life. In the human journey, the pursuit of relatively simple kind of goals that we as humans have gave rise to all of the complex behaviors that humans ultimately acquired and all of the knowledge that we eventually ended up with.

  7. 19:00

    So the kind of emergent complexity is much greater than the kind of starting point that was given. So it wasn't like evolution said, look, hey, humans, here's this idea of science and you need to kind of get to the most complex science that's out there. It was really something much simpler. So a way to describe this maybe would be perhaps that humans in the pursuit of our goals started coordinating a bit more.

  8. 19:26

    And then in the process of coordinating, we discovered that it was helpful to start communicating. And in the process of communicating, we started trading. And then in the process of trading, we discovered that basic arithmetic was useful. And then we started to discover that tools were useful. And then we discovered that some basic science was required to help understand how the tools would work.

  9. 19:47

    And that gave rise to more complex technology. And understanding the technology required more complex maths and so forth and so forth. So all in the pursuit of relatively simple goals, this incredible complexity emerged. So if we can get that process right, things are going to take off and lead to unbelievable things.