Coordinated LLM multi-agent systems for collaborative question-answer generation

• We propose CIR3, a novel framework for comprehensive and faithful QA generation. • The efficient information flow of CIR3 enables in-depth document analysis. • We employ transactive reasoning for deeper understanding in CIR3. • Our approach’s multi-perspective assessment ensures balanced views. • CIR3’s balanced collective convergence yields robust results. • CIR3 improves QA comprehensiveness (+23) and faithfulness (+17). Large Language Models (LLMs) excel at generating coherent and human-like questions and answers (QAs) across various topics, which can be utilized in various applications. However, their performance may be limited in domain-specific knowledge outside their training data, potentially resulting in low context recall or factual inconsistencies. This is particularly true in highly technical or specialized domains that require deep comprehension and reasoning beyond surface-level content. To address this, we propose C ollective I ntentional R eading through R eflection and R efinement ( CIR3 ), a novel multi-agent framework that leverages collective intelligence for high quality Question-Answer Generation (QAG) from domain-specific documents. CIR3 employs a transactive reasoning mechanism to facilitate efficient communication and information flow among agents. This enables for in-depth document analysis and the generation of comprehensive and faithful QAs. Additionally, multi-perspective assessment ensures that QAs are evaluated from various viewpoints, enhancing their quality and relevance. A balanced collective convergence process is employed to ensure that the agents reach a consensus on the generated QAs, preventing inconsistencies and improving overall coherence. Our experiments indicate a substantial level of alignment between the CIR3-generated QAs and corresponding documents, while improving comprehensiveness by 23 % and faithfulness by 17 % compared to strong baseline approaches. Code and data are available at https://github.com/anonym-nlp-ai/cirrr .