Real Alpha
Watch mode

Unsupervised Learning · 2 min instead of 59 min · Apr 9, 2026

Continual learning is what scaling GPT models was always for

Jakub Pachocki

Chief Scientist, OpenAI

  • Jacob EffronHost · Partner, Redpoint

    Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.

About this segment

Pachocki says he is confused by the continual learning movement. In his view, learning in context was the original promise of GPT models and it is what OpenAI is already building.

From “OpenAI’s Chief Scientist on Continual Learning Hype, RL Beyond Code, & Future Alignment Directions”, Apr 9, 2026. Watch the full episode

“I definitely agree that continual learning is really the thing. It's really the thing that we're building.”
Jakub Pachockiat 25:37 · Unsupervised Learning25:37

Takeaways

  1. GPT-3 was framed around learning to learn in context
  2. RL is used to teach models to learn in context more efficiently
  3. Scale is still the best path to continual learning
Watch 1:45 from 24:25

In this episode

Full episode
  1. 37:36Why OpenAI hides chains of thought to keep them monitorable5:56
  2. 12:28In-context learning may beat custom RL for most companies3:34

  3. 24:25Continual learning is what scaling GPT models was always for1:45

Up next · Lex Fridman PodcastPutting CUDA on GeForce nearly sank NVIDIA, but install base won4:13

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 24:29

    talk about was continual learning. And there was just, you know, it was in the zeitgeist. There's all these neolabs starting to go focus on continual learning. Some folks left OpenAI to go focus on that. I'm curious, like, you know, I think maybe part behind that is a belief that, like, you know, RL alone, you know, either won't get us there or will get us to, like, some level of very inefficient scaling. And it's kind of different than the way, you know, humans learn. I think even I've heard you say before, like, that, you know, RL is still very different today than the the way that humans learn.

  2. 24:57

    What's your take on that whole movement? Yeah, I am a little bit confused by it. Because in my mind, the whole excitement that we've had, even if you look at the titles of the GPT three paper, it is that, oh, this class of models is actually capable of continual learning. It's capable of learning to learn in context. That has been really the driving force, the kind of excitement to scale these GPT models further.

  3. 25:33

    That has been the premise for why we really need to teach them with RLs to learn in context more efficiently. And so I definitely agree that continual learning is really the thing. It's really the thing that we're building. But I don't really think this is a problem that's like, oh, So it's kind of ignored and off the path of what we're doing currently. I think it is what we're working towards. Yeah, in your mind, this is the single best path to get there is to continue to scale the pre-training in a row.

  4. 26:00

    I think that is how we've made the most progress on this problem so far. And I think there definitely are more ideas, more steps.