CMED: A Contrastive Diagnostic of Meaning-Preservation Metrics for Text Simplification

Abstract Automatic metrics are routinely used to judge whether text simplifications, including those produced by large language models, preserve the meaning of the original, yet it is unclear which kinds of meaning errors these metrics detect. Because simplification legitimately deletes, paraphrases and splits content, a trustworthy metric must penalize meaning-altering edits more than such legitimate changes. We introduce CMED (Contrastive Meaning-Error Diagnostic), a resource of controlled contrast sets derived from the ASSET test set. It pairs 718 human simplifications with rule-generated meaning-altering edits of seven types (negation, number, entity, antonym, quantifier, omission and insertion) and with legitimate changes (alternative human simplifications, deletion of peripheral content and sentence splitting). Following an analysis plan fixed before the test data were generated, we measured, for nine widely used or recently proposed metrics, how often a legitimate change is scored above a meaning-altering edit of the same simplification. The metrics showed complementary blind spots. Factual-consistency metrics based on natural language inference detected substitutions and insertions almost perfectly (0.94--0.99) but were at or below chance on omissions (0.41--0.51). BERTScore was near chance on number changes, and MeaningBERT also on negation and antonyms. Only a bidirectional entailment score exceeded 0.70 on every error type, yet it separated omissions of main assertions from legitimate peripheral deletions only weakly (0.64). The edits are synthetic, English and sentence-level, and their validity was checked automatically rather than by human annotators. CMED and its code are released to support error-type-resolved evaluation of simplification metrics.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11219172/v1
Primary Topic
Text Readability and Simplification
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

CMED: A Contrastive Diagnostic of Meaning-Preservation Metrics for Text Simplification

Emin Talip Demirkiran
1 links
Research Square
Text Readability and Simplification
preprint

CMED: A Contrastive Diagnostic of Meaning-Preservation Metrics for Text Simplification

Emin Talip Demirkiran
preprint en

Abstract

Abstract Automatic metrics are routinely used to judge whether text simplifications, including those produced by large language models, preserve the meaning of the original, yet it is unclear which kinds of meaning errors these metrics detect. Because simplification legitimately deletes, paraphrases and splits content, a trustworthy metric must penalize meaning-altering edits more than such legitimate changes. We introduce CMED (Contrastive Meaning-Error Diagnostic), a resource of controlled contrast sets derived from the ASSET test set. It pairs 718 human simplifications with rule-generated meaning-altering edits of seven types (negation, number, entity, antonym, quantifier, omission and insertion) and with legitimate changes (alternative human simplifications, deletion of peripheral content and sentence splitting). Following an analysis plan fixed before the test data were generated, we measured, for nine widely used or recently proposed metrics, how often a legitimate change is scored above a meaning-altering edit of the same simplification. The metrics showed complementary blind spots. Factual-consistency metrics based on natural language inference detected substitutions and insertions almost perfectly (0.94--0.99) but were at or below chance on omissions (0.41--0.51). BERTScore was near chance on number changes, and MeaningBERT also on negation and antonyms. Only a bidirectional entailment score exceeded 0.70 on every error type, yet it separated omissions of main assertions from legitimate peripheral deletions only weakly (0.64). The edits are synthetic, English and sentence-level, and their validity was checked automatically rather than by human annotators. CMED and its code are released to support error-type-resolved evaluation of simplification metrics.

Research Square
Eskisehir Technical University (TR)
Text Readability and Simplification
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon