First Experimental Study of the Proposed Unified Framework for Personalized Multidimensional Generative AI Evaluation of Short-Form Videos During Continuous Viewing

The increasing popularity of short-form video platforms has transformed digital media consumption into a continuous, personalized, and recommendation-driven experience. Although existing research has investigated video quality assessment, personalized recommendation, video summarization, and large language model (LLM)-based evaluation, these capabilities are generally treated as separate research problems. The previously proposed Personalized Multidimensional Generative AI Video Evaluation (PMGVE) framework addresses this fragmentation by introducing a unified AI-generated evaluation layer intended to accompany individual short-form videos during continuous viewing. This paper presents the first computational experimental study of the proposed framework, extending the previously published conceptual proposal through a controlled text-based proof-of-concept implementation. The experiment investigates whether a generative AI model can produce multidimensional video evaluations conditioned on different user profiles and whether the inclusion of social-comment information changes the generated assessments. Three synthetic user profiles, five representative video descriptions, and two experimental conditions were employed. Condition A provided the generative model with video descriptions and user profiles, whereas Condition B additionally supplied five synthetic comments associated with each video. A locally deployed Gemma 3 1B model generated 30 evaluation outputs. Three independent evaluator models—Qwen3 4B, Gemma 3 4B, and Qwen3 1.7B—subsequently assessed these outputs using six evaluation dimensions and an overall score, producing 90 independent evaluation records. All 30 generation tasks and all 90 evaluator tasks completed successfully. The generated video-quality scores exhibited substantial cross-user consistency: all Condition A outputs received a score of 70, while 14 of the 15 Condition B outputs also received 70, with one output receiving 75. In contrast, personalized recommendation scores demonstrated limited differentiation. Condition A produced scores ranging from 60 to 85, whereas all Condition B outputs converged to 85. The evaluator models also exhibited substantial differences in scoring severity, with mean overall scores of 1.467, 4.067, and 4.633 for Qwen3 4B, Gemma 3 4B, and Qwen3 1.7B, respectively. These findings demonstrate the computational feasibility of generating structured, user-conditioned evaluation outputs under the experimental conditions, while also identifying important limitations involving recommendation-score homogenization, evaluator disagreement, confidence-format inconsistency, and the absence of real audiovisual inputs and human evaluation. The results do not establish the effectiveness of PMGVE in real-world short-video platforms or demonstrate that its scores correspond to objective human judgments. Instead, they provide an initial empirical basis for refining the framework and designing subsequent multimodal and user-centered validation studies. To the best of the author's knowledge, this study constitutes the first experimental implementation and evaluation of the previously proposed PMGVE framework as a unified approach specifically integrating video-quality assessment, personalized recommendation scoring, personalized evaluative descriptions, and personalized semantic summaries within the context of continuous short-form video consumption.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24152765
Primary Topic
Recommender Systems and Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

First Experimental Study of the Proposed Unified Framework for Personalized Multidimensional Generative AI Evaluation of Short-Form Videos During Continuous Viewing

Jincheng Zhang
Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
preprint

First Experimental Study of the Proposed Unified Framework for Personalized Multidimensional Generative AI Evaluation of Short-Form Videos During Continuous Viewing

Jincheng Zhang
preprint en

Abstract

The increasing popularity of short-form video platforms has transformed digital media consumption into a continuous, personalized, and recommendation-driven experience. Although existing research has investigated video quality assessment, personalized recommendation, video summarization, and large language model (LLM)-based evaluation, these capabilities are generally treated as separate research problems. The previously proposed Personalized Multidimensional Generative AI Video Evaluation (PMGVE) framework addresses this fragmentation by introducing a unified AI-generated evaluation layer intended to accompany individual short-form videos during continuous viewing. This paper presents the first computational experimental study of the proposed framework, extending the previously published conceptual proposal through a controlled text-based proof-of-concept implementation. The experiment investigates whether a generative AI model can produce multidimensional video evaluations conditioned on different user profiles and whether the inclusion of social-comment information changes the generated assessments. Three synthetic user profiles, five representative video descriptions, and two experimental conditions were employed. Condition A provided the generative model with video descriptions and user profiles, whereas Condition B additionally supplied five synthetic comments associated with each video. A locally deployed Gemma 3 1B model generated 30 evaluation outputs. Three independent evaluator models—Qwen3 4B, Gemma 3 4B, and Qwen3 1.7B—subsequently assessed these outputs using six evaluation dimensions and an overall score, producing 90 independent evaluation records. All 30 generation tasks and all 90 evaluator tasks completed successfully. The generated video-quality scores exhibited substantial cross-user consistency: all Condition A outputs received a score of 70, while 14 of the 15 Condition B outputs also received 70, with one output receiving 75. In contrast, personalized recommendation scores demonstrated limited differentiation. Condition A produced scores ranging from 60 to 85, whereas all Condition B outputs converged to 85. The evaluator models also exhibited substantial differences in scoring severity, with mean overall scores of 1.467, 4.067, and 4.633 for Qwen3 4B, Gemma 3 4B, and Qwen3 1.7B, respectively. These findings demonstrate the computational feasibility of generating structured, user-conditioned evaluation outputs under the experimental conditions, while also identifying important limitations involving recommendation-score homogenization, evaluator disagreement, confidence-format inconsistency, and the absence of real audiovisual inputs and human evaluation. The results do not establish the effectiveness of PMGVE in real-world short-video platforms or demonstrate that its scores correspond to objective human judgments. Instead, they provide an initial empirical basis for refining the framework and designing subsequent multimodal and user-centered validation studies. To the best of the author's knowledge, this study constitutes the first experimental implementation and evaluation of the previously proposed PMGVE framework as a unified approach specifically integrating video-quality assessment, personalized recommendation scoring, personalized evaluative descriptions, and personalized semantic summaries within the context of continuous short-form video consumption.

Zenodo (CERN European Organization for Nuclear Research)
Recommender Systems and Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon