How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning

Summary · 5 min read instead of 2 h 1 m

Eric Bigelow is a member of technical staff at the interpretability startup Goodfire. He recently finished a Harvard psychology PhD with a dissertation titled Toward a Cognitive Science of Large Language Models. His work includes the forking paths paper on neural text generation and a follow-up, Forking Fast.

Bigelow argues that an LLM's decisions are determined largely by which token gets sampled, because the model then learns in context from what it just generated. His resampling work shows outcome distributions can collapse at single, sometimes arbitrary, tokens. He thinks reasoning models now work as a linearized search over possibilities, and that RL pushes chain of thought away from human-readable language. Bigelow says interpretability is badly underfunded relative to model building, and that agents like Goodfire's Silico can help if researchers keep their own scientific taste.

Minute by minute

1. 6:31

   Decisions are made when tokens are sampled. Bigelow describes his forking paths project on a GPT-3.5 model. He resampled about 30 rollouts at every token of a reasoning chain and recorded the final answers. Outcome distributions sometimes collapse suddenly, even at an arbitrary token such as an open parenthesis after kilowatt hours. He concludes that what a model says is largely set by the flip of a coin at each token.

2. 12:16

   Reasoning is in-context learning from sampled tokens. Bigelow says models learn in context from the words they have already generated. Each sampled token then pushes later reasoning to stay self-consistent with it. He notes that modern reasoning models do a linearized search and enumerate options, so their uncertainty dynamics look smoother. They still have forking points, and he calls reasoning and in-context learning neglected in interpretability.

3. 19:39

   In-context learning is broader than few-shot prompting. Bigelow defines in-context learning as everything a model does to adapt that does not happen in weights. That includes how representations shift as it reads each word, and how it updates beliefs while reading a story. He says prompting will not disappear however much fine-tuning is done. He also says aggregate curves look smooth while individual cases show sharp phase transitions, and that interpretability work often misses these dynamics.

4. 30:47

   Context engineering may beat per-user fine-tuning. Asked about continual learning, Bigelow expects current interpretability tools to carry over to fine-tuned models, perhaps with some probe fine-tuning. He bases this on shared low-dimensional structure in representations. His best guess is that harnesses, memories and context engineering will matter more than custom models for each person. He adds that memory is just tokens learned in context, which keeps it fairly interpretable.

5. 38:48

   Post-training narrows output diversity. Bigelow says supervised fine-tuning and further post-training reduce output diversity. RL shifts model behavior in a different direction again. He describes reasoning models as enumerating many possibilities, noting that DeepSeek R1 can say wait 50 times or more in one chain, and then picking one afterward. He thinks much of the raw capability was in pretrained models, with agent training mostly sharpening the distribution.

6. 46:33

   Seven to eight billion parameters is a sweet spot. Bigelow says he treats models from different developers as largely interchangeable stand-ins, and picks scale by compute and by whether the behavior shows up. He finds 7 to 8 billion parameters or more is where zero-shot behavior becomes interesting enough for interpretability. He says Kimi K3 was a step function for studying coding and reward hacking, which smaller models do not show.

7. 54:07

   Frontier interpretability leans on Chinese models. Asked whether outside researchers depend on Chinese models, Bigelow says that for now they do at the top scale, mainly because of coding ability. He thinks more US investment in open models could close the gap. He says restrictions on open models would slow study of emergent behaviors such as reward hacking, though foundational research could continue on older models.

8. 57:19

   Anthropomorphizing should be done with care. Bigelow describes pushback from cognitive scientists early in his PhD, and says it softened as models succeeded. He sees casual anthropomorphizing as fine when it helps understanding, as with intuitive theories of agents. He urges critics from psychology, linguistics and philosophy to engage on what beliefs and decisions are. He cites human rationalization of choices people never made as a parallel to forking paths.

9. 1:07:31

   Beliefs resemble Bayesian inference over latent concepts. Bigelow frames a model's beliefs as weights over latent concepts that context reweights, in the style of Bayesian models of cognition. He says LLMs let researchers test how beliefs are represented, which brains do not. He explains that uncertainty in his work means the spread of answers across resampled rollouts. He does not think models represent full future reasoning chains, though they may represent high-level paths ahead, as mathematicians sense a proof's shape.

10. 1:16:22

    Forking may be inevitable, so surface uncertainty. Bigelow thinks forking follows from wanting both output diversity and self-consistent reasoning. He wants interfaces that show users where an answer could branch, such as token probabilities or Janus's Loom, at the level of semantics. He says forking fast models smooth stretches and jumps statistically because resampling is expensive. He says the stochastic parrot idea should be retired but stochasticity is not dead, even in looped models.

11. 1:31:44

    RL gives no reason to keep chain of thought readable. Bigelow says outcome-based RL has no incentive to keep chains of thought or sub-agent messages human readable. He compares this to language evolution and to Goodhart's law, where optimizing monitorability can make chains less faithful. He cites the stolen chain of thought paper, where GPT models discuss musicals in a spy-like way. He used to be bullish on chain of thought monitoring and has shifted.

12. 1:41:54

    Interpretability needs far more investment. Bigelow's answer to how to understand critical-token decisions is more interpretability research. He contrasts trillion-dollar model builders with a field where, he says, Goodfire may be the only substantial company valued above a billion dollars. He calls academia underfunded. He counts behavioral evaluations as part of understanding AI and says tools must scale efficiently.

13. 1:45:52

    Silico works best with human scientific taste. Bigelow advises against handing agents a whole project. He recommends giving context, explaining why results look wrong and keeping the scientific narrative yourself. He says Silico did all the research for the Forking Fast paper within a week, but he spotted the smooth curves that led to the statistical model. He likens Silico to an eager first- or second-year PhD student.

Published Oct 10, 2026.