Match model effort to the task and ask for decision notes

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 25:15

   if you want, for like, we see the effort scales with basically the complexity of the task. So for security, effort gets like way more results. Like high effort versus like low effort gets like changes the evals a lot. But for software engineering, it doesn't change a huge amount because effort is mostly spent on the verification and the like edge case testing and things like that. And so like being able to like give the model that guidance of like, hey, this problem is something that I think I want you to spend a lot of time verifying and edge case testing.

2. 25:49

   How about model in the mix? So there's Opus and Fable with effort. There's also Haiku in there. Yeah, yeah, yeah. It's not quite true yet, but it's very close. I think the frontier models will be Pareto dominant over almost everything. And sometimes I think Opus might be Pareto dominant. Do you know what I mean? I think depending on how things shake out, if it's a newer version of Opus. But I think that increasingly it's just going to be like the smart model is going to be able to do the simple task for less tokens than the other models, basically, because of verification.

3. 26:30

   With verification, in the limit, your model doesn't need to verify. If it's a perfect model, it just does the work once and it's like, okay, I did it. you know and increasingly with Fable I'm like dude you don't need to spin up Chromium and screenshot all of these things like I see it like you did it right and so a lot of the at higher effort you spend more of those tokens verifying but if you're working on simpler problems and a lot of software engineering is like you know like well like in Fable like low and medium's ability it can spend less tokens verifying and as the models get smarter and smarter they will just be able to like alright done you know I can run the lint for sanity's sake but I know it lints.

4. 27:14

   You don't even need to do that. And that will be so much more token efficient than the smaller models basically. Is there a good practice on our side that we can use to see if we're using too much effort? I freaking hate wasting time on that kind of stuff. I know what you mean. In this blog post my rough distribution is code review and security should be like higher max, basically. And like software engineering, settings per domain.

5. 27:43

   Yeah, I think like if you're doing like UI or something like that, like low. And medium, I think is you're building like an API and you want to make sure like you cover enough edge cases, you know? And so I think building, like I said, that mental model of like how things work across these distributions is like, yeah, part of the job. This is more intuition driven or evil? Because I'm guessing this would change.

6. 28:06

   He has evils. Yeah, so what I did in the blog post is I go over all of the terminal bench evals, basically. So there are like 70 problems. And I show that like, okay, you know, like in the security problems, it does more. And then I also like look at some of the transcripts just in terms of like, how, what does it answer? What does it forget or something? And a lot of times, this is another prompting tip I have, is asking it to make decision notes or implementation notes.

7. 28:36

   Because in basically every eval problem that it faces, it thinks about the correct solution and decides not to do it. It's like, oh, here's the answer. What if I did this? And then it's like, oh, probably not. And then keeps going. and this is like the majority of the failures you know what I mean at like a higher max level it's very rare that the model just doesn't know how to do something if you just have these implementation notes then you can review and you can be like oh actually I want you to do this thing that you didn't do the models are getting better at surfacing that overall like I see in the transcripts that Fable 5.1 like when it does this output it will call out its decision making as well but making this more explicit in the harness is better.

8. 29:24

   And now we're allowing ways of you modifying the harness so you can add some capabilities there.