Academics should take big bets that industry labs won't

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:16

   students or like in academia comes when people care about problems that maybe like most people in industry are not looking at. I think this is the issue that like a lot of grad students work on things that benefit, like that look good to an industry lab. Like, for example, they'll work on some benchmark. It's really popular now. I mean, I think benchmarks in 2023 were a very different story than benchmarks now.

2. 15:41

   There are a lot of people that work on harnesses and meta-harnesses and specific harnesses for XYZ tasks. And when you really think about it, the reason someone would work on this is, like, maybe there's, like, a clear goal shaped around the models that we have today of, like, this is what I want to see. But, like, I'll give the, like, RLM, like, the recursive language model paper as an example because I think, like, it's a super, super simple idea.

3. 16:10

   I think when it came out as well, like, there were a lot of people that were, like, when they see something like that, they're, like, what is even the purpose of this? This is too cool. Yeah. Like why? Like what? This is just sub agents or something. Right. And I think it's like when you get a reaction like that, it's almost like a good sign in the sense that, like, it's clear that people aren't thinking about what the purpose of this is.

4. 16:32

   And I will give another example of like Sweebench. When Sweebench came out, I mean, Ophir loves to tell this story. When it came out, like nobody cared. Like everybody was like, this is an impossible task. Like why would we ever even consider this as a benchmark? And it wasn't until Devin came out that everyone was like, whoa, like this is something we want to hill climb. And I think this rings true for you tend to see that a lot of ideas.

5. 16:56

   I think like my favorite, I guess, example of this is Eric Seligman's work. with like Star and like Quiet Star. Like I think when you read the paper, at least when I first read the paper, I was like, is this not like an obvious idea? Or maybe not, I don't know. I mean, I was like, oh, this seems really simple or like Chain of Thought and it's the same thing or like Chunyu's React. It's like, okay, like, yeah, I mean, sure.

6. 17:20

   But then like when you really think about it, it's like, why, what is the value of the paper? And I think it comes from like, it tells a bit of a story as to like what you want the field to look like. And that is something that it's very hard to do this in academia, because if you look at all these papers, QuietStar, React, RLMs, SweetBench, none of these papers are, it's not like a GPT-6 Astra release.

7. 17:45

   You know, it's not like everyone's like, oh, my gosh, like, I'm going to use this now. This is the best thing in the world. Like, academia just can't afford to do this, at least right now. I mean, there's a whole slew of reasons why I think that should change. But I think it's like, if you don't have, as a PhD student, I think you're in such a unique position where you can work on literally whatever you want, for the most part, if you're not taking advantage of that and working on things that like nobody cares about, or like people see as some trivial thing, like, oh, I thought about this, but like, I don't use it. I just think like in the end, the research is just never going to be that interesting, because you kind of need to take big bets if you're going to be in academia. Because otherwise, I think like, just go to an

8. 18:24

   industry lab, like, you know, they have tons of resources, tons of talent. Why constrain yourself in an area where you don't have a lot of resources and like there's not even that many people around. And I think it's just, it literally just comes down to like big bets. You just have to take big bets and like a lot of them will fail, you know? Like that's just, it's natural. But I think that is, as a PhD student, like that's the biggest advantage you have over any single person at another lab because you don't have to deal with bureaucracy