RL for materials works by rewarding sub-steps, not whole discoveries

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:26

   what to make. So that's the stability of the atoms together, as well as do we expect it to have the properties we want. So we don't just want a novel configuration of the atoms, we want them when put together to have the property of interest. Next is, how do you synthesize it? How do you actually make the thing? That's also non-trivial. So even if you know something will hold together, the process of actually, you know, what are the processing conditions to actually bring that forth is highly non-trivial. But again, like the thing, once you've done those two steps and you've made something, it doesn't come off labeled. So you actually have to characterize it and figure out what you made. So in that case, rather than thinking about the RL environment of, we're just going to kick off an experiment, wait a couple of days and try to do like a update on like, did you

2. 10:12

   find a room temperature superconductor or not? That's completely infeasible. I was going to ask that yeah like you can't leave your gpu sitting for a week or something yeah it's completely infeasible because you don't have enough agents to reduce the variance sufficiently the rollout time is going between the different instruments you're waiting there's just some physics limits of the experiments um for like the actual like synthesis or furnace time uh so it's just too too slow too noisy so the way we think about the ai program is basically constructing agents around this set of data that we've produced and so in the case of characterization you're looking for rl environments that are rewarding the identification of the phases actually present are you able to um given the raw experimental data fit that effectively so the way we do this is we'll

3. 11:03

   shoot x-rays um at the material uh x-rays have a frequency or a wavelength roughly measurable or commensurate to the spacing between atoms so you get nice diffraction patterns that kind of give a fingerprint of the crystal structure. And doing this identification of what actually is present is highly non-trivial. And so some of our early AI work has been on basically producing systems to do this. And this allows us to move much more quickly too, because if you're running so many experiments, you really quickly become bottlenecked on your ability to understand what was produced.

4. 11:38

   But again, now the reinforcement learning environment is fairly straightforward, which is you're going to reward the identification of the phases present. You're going to penalize spurious phases or things that don't have any like real chemical plausibility. So that's like a, you know, a little sustained thing where you can kind of do this for the different pieces. And when you stitch this together, that's...

5. 11:59

   the end-to-end discovery loop. Maybe another piece would be, as we build up a large basket of experimental data, you can think of another thing of basically timestamping the state of the world. So you could say, at this date, this was our experimental evidence up until that point. And you can construct reinforcement learning environments where you say, okay, given that body of experimental evidence and given the choices, what was like the next choice that the scientists made or what was an outcome of that experiment.

6. 12:34

   And this is really interesting too because this allows us to do types of programs that would be harder externally because if the pre-trained model has already memorized some of this data and then you try to do reinforcement learning on that, if it already knows the answer and you construct reinforcement learning tasks that depend on that answer, it can fake work. So, and then, I mean, because it's like, oh, well, it already knows the answer.

7. 13:00

   So it doesn't actually have to do the hard physics reasoning. It doesn't have to actually do the calculation simulations. And it gets to the right answer. And then you reinforce that policy and you upweight those reasoning strategies. Those reasoning strategies will not generalize to novel systems. So it's not useful to us. But as we build up baskets of experimental data and like stitching things together, having the lineage, the whole process, that gives us the ability to construct new types of environments that are just not plausible elsewhere. I want to ask my question, but I'm worried that I'm taking