Make agents prove a bug exists and prove the fix

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:43

   Now, in the process of building, there is this proof step. And this skill, it only works for some machines, right? I think if you have a Linux machine with FFMPEG and all that stuff installed, basically what this does is it captures the before state while reproducing the issue prior to fixing it, and then it will show the after state. This works great with cursor cloud agents. And I can show an example of what this is.

2. 11:16

   Let me just pull this up. Yeah. So what this skill allows with like a cursor or a Devon Cloud agent is it basically forces, not forces, but it tells my agent to actually produce evidence of said bug not working and said bug working. Right. So it records a before and after state. And if I were to show you the skill, it's a bunch of scripts, as you will see. And then just to people who don't know, FFMPEG is basically just like a video recording.

3. 11:51

   Basically, right? So if you have a Linux Mac OS, I haven't tested a Windows one. I don't have a Windows machine, unfortunately. So this is why I say this skill might work, might not work. But basically with a cloud agent, this is great because I'm getting video evidence of the agent testing my skill out. This works great for bugs because, for example, for some of the projects I'm working on where we actually have users, a product person will say, oh, we found this bug and leave a linear ticket.

4. 12:19

   And what happens now is on linear, I wish I could show, but some of it is private information. With linear, you can assign work to a person, but you can also assign work to an agent. Right. And basically what this skill does is it forces the agent to prove that this bug actually happens. Right. Because sometimes it just might have been user error. Sometimes it just something might have went down. Right. We want to make sure that we prove that a bug is actually a bug.

5. 12:47

   And what this skill also does is it proves the fix as well. Right. I think in this case, this is proving that it can get a response and stream properly. Right. And it streams properly. There is a streaming issue. But let's say that doesn't work for you. There's another skill called before and after. This is a skill I stole from Vercel. So shout out to the Vercel team for this. And basically what this skill does is it takes a before and after screenshot of said feature build.

6. 13:14

   And I can show an example of this so people can have an idea. Let me go to a project I've been working on and just pull up one. Maybe let's try this right here. So this is a great example. I think this is a tracking feature for email delivery that I was working on. And before, this is existing mailbox. And then the after, it shows the result that, oh, whatever file this was malware, right? So I get a before and after screenshot sort of showing the change that was made.

7. 13:53

   But for, let's say, for a feature that's not really UI specific or doesn't have like, you know, something like UI to screenshot, then what the skill will do. And this is pretty cool. Performance. I just felt performance wrong in front of thousands of people. I apologize. And for something that's not screenshotable, it will give you numbers. Right. So in this case, this was the dashboard I had built using GPT 5.6 Sol.

8. 14:21

   The navigation was just terribly slow. And I asked Astra to do a performance update, and it got it from 850 milliseconds to 60, 817 to 61. You could see considerable performance update, but this isn't necessarily something you could show with a screenshot. So that's what before and after will do, right? Again, proving the work of the agent. So, okay. So evidence-driven testing is it records a video of the main branch without the new feature and the work tree branch.

9. 14:53

   So it kind of compares the two. Yeah. So basically if it more so the before is more so useful if a bug was reported, right? So say a bug was reported, what it will do is it will like reproduce the bug, right? So let's say there was a streaming bug when you refresh the app after you sent the message, right? The moments. So it will record that, right? If there is no bug per se, there's no before for it to record. It will just record the after state, right? So evidence-driven testing is really good for an app that's already in production and you're getting a bug report. But before and after always works. And I find this to be very, very great because it'll show you the improvements made, right?