HybridEval: An Improved Novel Hybrid Metric for Evaluation of Text Summarization

The present work re-evaluates the evaluation method for text summarization tasks. Two state-of-the-art assessment measures e.g., Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and Bilingual Evaluation Understudy (BLEU) are discussed along with their limitations before presenting a novel evaluation metric. The evaluation scores are significantly different because of the length and vocabulary of the sentences, this suggests that the primary restriction is its inability to preserve the semantics and meaning of the sentences and consistent weight distribution over the whole sentence. To address this, the present work organizes the phrases into six different groups and to evaluate “text summarization” problems, a new hybrid approach (HybridEval) is proposed. Our approach uses a weighted sum of cosine scores from InferSent’s SentEval algorithms combined with original scores, achieving high accuracy. HybridEval outperforms existing state-of-the-art models by 10-15% in evaluation scores.

Authors

Institutions

Publication Details

Journal
Journal of Informatics and Web Engineering
Published
2024-10-16
DOI
https://doi.org/10.33093/jiwe.2024.3.3.15
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

HybridEval: An Improved Novel Hybrid Metric for Evaluation of Text Summarization

Muhammad A. B. Fayyaz, Suppawong Tuarob, Jia Zhu, Pin Shen Teh et al.
Journal of Informatics and Web Engineering
Topic Modeling
article

HybridEval: An Improved Novel Hybrid Metric for Evaluation of Text Summarization

Muhammad A. B. Fayyaz, Suppawong Tuarob, Jia Zhu, Pin Shen Teh, Naif Radi Aljohani, Bilal Ahmad, Ali Ansari, Farooq Zaman, Saeed‐Ul Hassan, Tipajin Thaipisutikul, Raheel Nawaz, Raheem Sarwar
article en

Abstract

The present work re-evaluates the evaluation method for text summarization tasks. Two state-of-the-art assessment measures e.g., Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and Bilingual Evaluation Understudy (BLEU) are discussed along with their limitations before presenting a novel evaluation metric. The evaluation scores are significantly different because of the length and vocabulary of the sentences, this suggests that the primary restriction is its inability to preserve the semantics and meaning of the sentences and consistent weight distribution over the whole sentence. To address this, the present work organizes the phrases into six different groups and to evaluate “text summarization” problems, a new hybrid approach (HybridEval) is proposed. Our approach uses a weighted sum of cosine scores from InferSent’s SentEval algorithms combined with original scores, achieving high accuracy. HybridEval outperforms existing state-of-the-art models by 10-15% in evaluation scores.

Journal of Informatics and Web EngineeringVol. 3(3)
Manchester Metropolitan University (GB), Information Technology University (PK), Zhejiang Normal University (CN), Gulf University for Science & Technology (KW), King Abdulaziz University (SA), University of Staffordshire (GB), Mahidol University (TH)
Quality Education
Openalex Percentile: Top 84%
Topic Modeling
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.