Evaluating multi-LLM pipelines for early-stage software requirements elicitation

Abstract Requirements elicitation is both economically and cognitively costly, often hampered by incomplete domain knowledge and the “blank-page effect,” particularly in contexts where stakeholders have limited understanding of the problem space. To support analyst productivity and elicitation completeness, we explore how Large Language Models (LLMs) can generate an initial requirements baseline from informal system descriptions, providing a structured starting point for stakeholder interaction. We evaluate an engineered multi-LLM iterative pipeline in which complementary models, strict prompt contracts, and Generate–Unify–Improve consolidation rules jointly produce and refine candidate functional requirements. Each model contributes from its perspective, analogous to consulting several domain experts. The pipeline combines candidate requirements produced by multiple models and iteratively consolidates, extends, and refines the resulting requirement set. To assess the effectiveness of this orchestration pipeline, we conduct two complementary experiments. The first compares single-LLM and multi-LLM setups, both with and without iterative refinement stages, against ground-truth requirements extracted from public Software Requirements Specifications. Building on the findings of the first experiment, the second experiment introduces human evaluators who assess the generated requirements for clarity, usefulness, and completeness. Finally, we let an AI-powered coding assistant produce a system prototype based on the generated requirements: then humans assess the prototype features for usefulness and completeness. Together, these evaluations assess ground-truth alignment through similarity metrics and explore perceived practical usefulness through a human-centered study.