CriticalBrew at CQs-Gen 2025: Collaborative Multi-Agent Generation and Evaluation of Critical Questions for Arguments

This paper presents the CriticalBrew submission to the CQs-Gen 2025 shared task, which focuses on generating critical questions (CQs) for a given argument.Our approach employs a multi-agent framework containing two sequential components: 1) Generation: machine society simulation for generating CQs and 2) Evaluation: LLM-based evaluation for selecting the top three questions.The first models collaboration as a sequence of thinking patterns (e.g., debate reflect).The second assesses the generated questions using zero-shot prompting, evaluating them against several criteria (e.g., depth).Experiments with different open-weight LLMs (small vs. large) consistently outperformed the baseline, a single LLM with zero-shot prompting.Two configurations, agent count and thinking patterns, significantly impacted the performance in the shared task's CQ-usefulness evaluation, whereas different LLM-based evaluation strategies (e.g., scoring) had no impact.Our code is available on GitHub 1 .