Coding Agents Lunch & Learn Session 24: Understanding Uncertainty in LLM Reasoning

Join us for Session 24 of the Coding Agents Lunch & Learn, our weekly community gathering for anyone interested in coding agents, agentic AI, AI-powered developer tools, and the rapidly evolving world of software development.

This week, we’re joined by Eric Bigelow from Goodfire to explore a fascinating question at the heart of how we understand modern reasoning models: how can we better understand the uncertainty behind an LLM’s reasoning without requiring an enormous amount of computation?

LLM reasoning is inherently stochastic. Given the same question, a model can produce different reasoning chains, and understanding that distribution can reveal important information about how models arrive at their answers.

Eric will walk us through his recent research on making resampling-based analysis of LLM reasoning more computationally efficient. The work looks at how uncertainty evolves throughout generated reasoning chains, showing that when enough reasoning chains are sampled, uncertainty dynamics tend to converge to stable patterns, while much of the apparent noise comes from the sampling process itself rather than the model being highly sensitive to every individual token or reasoning step.

The research also introduces a statistical model for smoothing noisy, low-sample rollout data, allowing researchers to approximate the insights of much larger sampling runs while significantly reducing the computational cost.

This will be a loose, conversational session rather than a tightly structured presentation. Eric will give us an overview of the research and its different pieces, followed by an open discussion with the hosts and community.

We’ll dig into questions around LLM uncertainty, reasoning chains, resampling, sampling efficiency, interpretability, and what these techniques can tell us about how reasoning models actually behave.

Whether you’re building coding agents, researching LLMs, working with AI developer tools, or simply curious about how we can better understand increasingly capable reasoning systems, come join us to learn from the research, ask questions, and explore the ideas together.

Bring your lunch, bring your questions, and come hang out with us!