Beyond AI Response Evaluation: Dual-Level User Message and Interaction Scoring for Generative Artificial Intelligence — A Unified Framework for Multidimensional, Context-Aware, and Longitudinal Evaluation of User Contributions (DUMIS-GAI)

The evaluation of generative artificial intelligence (GenAI) systems has predominantly concentrated on the quality, accuracy, relevance, helpfulness, and safety of AI-generated responses. Although substantial progress has been made in automated dialogue evaluation, large language model (LLM)-as-a-judge methodologies, and conversational quality assessment, comparatively less attention has been directed toward treating the user's own contributions as independent and explicitly assessable objects within generative AI interactions. This paper introduces Dual-Level User Message and Interaction Scoring for Generative AI (DUMIS-GAI), a unified conceptual framework that establishes two complementary evaluation levels: message-level user scoring and interaction-level user scoring. Message-level scoring enables a generative AI system to evaluate an individual user-authored message, question, instruction, response, or contribution against configurable multidimensional criteria. Interaction-level scoring evaluates a complete user–AI exchange or a selected interaction segment as an integrated communicative process. Unlike conventional AI response evaluation, the proposed framework explicitly positions the user as the subject of evaluation rather than merely treating user messages as inputs to be processed. The framework incorporates contextual interpretation, configurable scoring dimensions, weighted aggregation, evidence-linked explanations, user-controlled evaluation, temporal tracking, historical comparisons, and privacy-preserving score management. It further distinguishes communicative quality from factual correctness, task difficulty, language proficiency, personal characteristics, and subjective preferences to reduce inappropriate or misleading judgments. A formal scoring model, operational architecture, evaluation methodology, governance principles, and research propositions are developed. To the best of our knowledge, this paper proposes one of the first unified, generative-AI-oriented conceptual frameworks specifically organized around both independently scoring individual user messages and evaluating complete user–AI interactions as two connected levels of user-centered assessment. The framework provides a foundation for transforming generative AI from a system that primarily evaluates its own outputs into an interactive environment capable of transparently evaluating, explaining, and tracking the quality of user contributions over time.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24273556
Primary Topic
Speech and dialogue systems
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Beyond AI Response Evaluation: Dual-Level User Message and Interaction Scoring for Generative Artificial Intelligence — A Unified Framework for Multidimensional, Context-Aware, and Longitudinal Evaluation of User Contributions (DUMIS-GAI)

Jincheng Zhang
8 links
Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
preprint

Beyond AI Response Evaluation: Dual-Level User Message and Interaction Scoring for Generative Artificial Intelligence — A Unified Framework for Multidimensional, Context-Aware, and Longitudinal Evaluation of User Contributions (DUMIS-GAI)

Jincheng Zhang
preprint en

Abstract

The evaluation of generative artificial intelligence (GenAI) systems has predominantly concentrated on the quality, accuracy, relevance, helpfulness, and safety of AI-generated responses. Although substantial progress has been made in automated dialogue evaluation, large language model (LLM)-as-a-judge methodologies, and conversational quality assessment, comparatively less attention has been directed toward treating the user's own contributions as independent and explicitly assessable objects within generative AI interactions. This paper introduces Dual-Level User Message and Interaction Scoring for Generative AI (DUMIS-GAI), a unified conceptual framework that establishes two complementary evaluation levels: message-level user scoring and interaction-level user scoring. Message-level scoring enables a generative AI system to evaluate an individual user-authored message, question, instruction, response, or contribution against configurable multidimensional criteria. Interaction-level scoring evaluates a complete user–AI exchange or a selected interaction segment as an integrated communicative process. Unlike conventional AI response evaluation, the proposed framework explicitly positions the user as the subject of evaluation rather than merely treating user messages as inputs to be processed. The framework incorporates contextual interpretation, configurable scoring dimensions, weighted aggregation, evidence-linked explanations, user-controlled evaluation, temporal tracking, historical comparisons, and privacy-preserving score management. It further distinguishes communicative quality from factual correctness, task difficulty, language proficiency, personal characteristics, and subjective preferences to reduce inappropriate or misleading judgments. A formal scoring model, operational architecture, evaluation methodology, governance principles, and research propositions are developed. To the best of our knowledge, this paper proposes one of the first unified, generative-AI-oriented conceptual frameworks specifically organized around both independently scoring individual user messages and evaluating complete user–AI interactions as two connected levels of user-centered assessment. The framework provides a foundation for transforming generative AI from a system that primarily evaluates its own outputs into an interactive environment capable of transparently evaluating, explaining, and tracking the quality of user contributions over time.

Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon