Comparing Methods for Multi-Label Classification of Manipulation Techniques in Ukrainian Telegram Content

Detecting manipulation techniques in online text is vital for combating misinformation, a task complicated by generative AI.This paper compares machine learning approaches for multi-label classification of 10 techniques in Ukrainian Telegram content (UNLP 2025 Shared Task 1).Our evaluation included TF-IDF, fine-tuned XLM-RoBERTa-Large, PEFT-LLM (Gemma, Mistral) and a RAG approach (E5 + Mistral Nemo).The fine-tuned XLM-RoBERTa-Large model, which incorporates weighted loss to address class imbalance, yielded the highest Macro F1 score (0.4346).This result surpassed the performance of TF-IDF (Macro F1 0.32-0.36), the PEFT-LLM (0.28-0.33) and RAG (0.309).Synthetic data slightly helped TF-IDF but reduced transformer model performance.The results demonstrate the strong performance of standard transformers like XLM-R when appropriately configured for this classification task.

Authors

Institutions

Publication Details

Published
2025-01-01
DOI
https://doi.org/10.18653/v1/2025.unlp-1.5
Primary Topic
Text and Document Classification Technologies
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Comparing Methods for Multi-Label Classification of Manipulation Techniques in Ukrainian Telegram Content

Оleh Melnychuk
Text and Document Classification Technologies
article

Comparing Methods for Multi-Label Classification of Manipulation Techniques in Ukrainian Telegram Content

Оleh Melnychuk
article en

Abstract

Detecting manipulation techniques in online text is vital for combating misinformation, a task complicated by generative AI.This paper compares machine learning approaches for multi-label classification of 10 techniques in Ukrainian Telegram content (UNLP 2025 Shared Task 1).Our evaluation included TF-IDF, fine-tuned XLM-RoBERTa-Large, PEFT-LLM (Gemma, Mistral) and a RAG approach (E5 + Mistral Nemo).The fine-tuned XLM-RoBERTa-Large model, which incorporates weighted loss to address class imbalance, yielded the highest Macro F1 score (0.4346).This result surpassed the performance of TF-IDF (Macro F1 0.32-0.36), the PEFT-LLM (0.28-0.33) and RAG (0.309).Synthetic data slightly helped TF-IDF but reduced transformer model performance.The results demonstrate the strong performance of standard transformers like XLM-R when appropriately configured for this classification task.

National University "Kyiv Aviation Institute" (UA)
Openalex Percentile: Top 90%
Text and Document Classification Technologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.