Developing Multi-Agent LLM Applications Through Continuous Human-LLM Co-Programming

The rapid advancement of Large Language Models (LLMs) has opened new possibilities for intelligent multi-agent systems capable of autonomously performing complex tasks. To build such systems, LLMs can be leveraged for task-solving, tool interaction, and code generation but at the same time their costs and unpredictability have to be properly managed. To do so this paper introduces COPMA, a model-based approach to enabling continuous human-LLM co-programming of multi-agent LLM applications. COPMA uses feature-block models to track application features and their implementations as agents and code blocks. Supported by co-programming patterns, de-velopers are guided in constructing, refining, and refactoring feature implementations via trial-and-errors with LLM agents, leveraging their feedback, suggestions, and code examples. The patterns guide the shift of feature implementations between agents and code to balance flexibility, predictability, and cost. Our experience in developing LLM agents for collecting and reviewing medical research papers demonstrates that human-LLM co-programming can reduce development effort to enable rapid prototyping of multi-agent LLM applications.