Developing LLM-based multi-agent systems in software engineering: a mixed-method experience report

Abstract The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, by means of a mixed-method study, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. We qualitatively analyzed eight representative LLM-based multi-agent frameworks selected from an initial pool of 20 publicly available frameworks, and quantitatively evaluated them using the software engineering scenario. The qualitative analysis reveals a trade-off between low-code frameworks, which emphasize ease of use through integrated agent creation, orchestration, and monitoring capabilities, and high-code frameworks, which provide greater flexibility and extensibility for advanced applications. The quantitative evaluation shows that all frameworks achieve comparable recommendation effectiveness, whereas their efficiency differs considerably: Semantic Kernel Chat incurs the highest token consumption, while AutoGPT exhibits the longest execution time. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.