From Code Completion to Autonomous Software Engineering Agents // Kilian Lieret // Agent Hour

As language models have advanced, they have moved beyond code completion and are beginning to tackle software engineering tasks in a more autonomous, agentic way. However, evaluating agentic capabilities is challenging. To address this, we first introduce SWE-bench, a benchmark built from real GitHub issues that has become the standard for assessing AI’s ability to resolve complex software tasks in large codebases. We will discuss the current state of the field, the limitations of today’s models, and how far we still are from truly autonomous AI developers. Next, we will explore the fundamentals of agents based on SWE-agent, a simple yet powerful agent framework designed for software engineering but adaptable to a variety of domains. By the end of this talk, you will have an understanding of the current frontier of agentic AI in software engineering, the challenges ahead, and various tips and tricks on optimizing AI agents for tool use and iterative problem solving of reasoning-heavy tasks.