Towards Log Analysis for Reliability Engineering of Agentic Systems

Large Language Models have enabled agent-based approaches that can autonomously perform complex tasks by interacting with external tools and environments. By autonomously chaining multiple tool calls, these agentic systems can solve sophisticated problems that extend beyond the generated text. However, unlocking such extended capabilities comes with an increased risk of unexpected behaviors, failures, and safety violations. As a result, enhanced observability is required to gain actionable insights into the operation of agentic systems from runtime-generated data, enabling the analysis of their behavior and the identification of potential issues, thereby ensuring their reliability and safety. In this vision paper, we propose a research agenda towards log analysis for the reliability engineering of agentic systems. Logs have historically been a cornerstone for monitoring, analyzing, and improving the operation of software systems. However, the unique characteristics of agentic systems require a fundamental re-engineering and redesign of both analysis units and techniques to effectively leverage logs for reliability engineering. We outline four specific research streams aiming to systematically transpose the well-established log analysis pipeline from conventional software systems to agentic systems.