SpecEvaluator enables structured multi-expert quality assessment of speech dialogue systems with specialized large audio models

We investigate automatic evaluation for spoken dialogue systems and propose SpecEvaluator, a structured multi-dimensional evaluation framework. Existing methods either rely on ASR transcription or produce a single holistic score, which hinders the preservation and discrimination of key aspects of speech dialogue such as semantic content, emotional expression, and speech quality. To address this, SpecEvaluator decomposes the evaluation task into three dimensions—content coherence, emotional appropriateness, and speech quality—each judged by a dimension-specific audio-language model. A zero-shot aggregation module then generates the final score and explanation. We further construct a multi-dimensional evaluation dataset with controllable quality gradients and generate dimension-level supervision signals using a teacher model. Experiments demonstrate that SpecEvaluator consistently outperforms strong baselines across ranking accuracy, correlation, and grade classification.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-20
DOI
https://doi.org/10.1038/s41598-026-79430-0
Primary Topic
Speech and dialogue systems
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

SpecEvaluator enables structured multi-expert quality assessment of speech dialogue systems with specialized large audio models

Dan Qu, Hao Zhang, Ruopan Wang
Scientific Reports
Speech and dialogue systems
article

SpecEvaluator enables structured multi-expert quality assessment of speech dialogue systems with specialized large audio models

Dan Qu, Hao Zhang, Ruopan Wang
article en

Abstract

We investigate automatic evaluation for spoken dialogue systems and propose SpecEvaluator, a structured multi-dimensional evaluation framework. Existing methods either rely on ASR transcription or produce a single holistic score, which hinders the preservation and discrimination of key aspects of speech dialogue such as semantic content, emotional expression, and speech quality. To address this, SpecEvaluator decomposes the evaluation task into three dimensions—content coherence, emotional appropriateness, and speech quality—each judged by a dimension-specific audio-language model. A zero-shot aggregation module then generates the final score and explanation. We further construct a multi-dimensional evaluation dataset with controllable quality gradients and generate dimension-level supervision signals using a teacher model. Experiments demonstrate that SpecEvaluator consistently outperforms strong baselines across ranking accuracy, correlation, and grade classification.

Scientific Reports
PLA Information Engineering University (CN)
Science and Technology Department of Henan Province
Speech and dialogue systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon