RAE: A Role-Based Adaptive Framework for Evaluating Automatically Generated Public Opinion Reports

Public Opinion Reports are essential tools for crisis management, yet their evaluation remains a critical bottleneck that often delays response actions. Recently, dominant Large Language Model (LLM)-based evaluators often overlook a critical challenge: highly open-ended dimensions such as “innovation” and “feasibility” require synthesizing diverse stakeholder perspectives, as different groups judge these qualities from fundamentally different perspectives. Motivated by this, we propose the Role-based Adaptive Evaluation (RAE) framework. This framework employs an adaptive mechanism leveraging multi-perspective evaluation insights through role-based analysis, and further introduces dynamically generated roles tailored to specific contexts for these dimensions. RAE further incorporates multi-role reasoning aggregation to minimize individual biases and enhance evaluation robustness. Extensive experiments demonstrate that RAE significantly improves alignment with human expert judgments, especially on challenging highly open-ended dimensions.