Systematic subjective evaluation of contemporary generative AI models: challenges and methodology advancements

Deploying advanced generative AI models presents significant challenges, primarily due to the lack of accurate objective metrics for evaluating and monitoring model quality. As a result, human subjective evaluation becomes crucial for assessing model performance and guiding development. However, subjective-evaluation has multiple limitations, including the expensive nature of subjective data collection and the time-consuming effort needed to process raw data to minimize bias/outliers. These limitations are magnified in projects that require multiple iterations. This paper focuses on a case-study of evaluating the quality of GenAI-powered lipsynchronized autodubbing. To address the limitations of conventional approach, we employ state-of-the-art SUREAL analysis to assess the statistical significance of subjective test results. Furthermore, we enhance our methodology with two-way random effects analysis and hierarchical bootstrap to refine confidence intervals. These improvements enable efficient and effective subjective quality evaluation and thus accelerate genAI model development. This paper also identifies future opportunities for improvement, aiming to create a more efficient and reliable evaluation process.