BanglishAIGT: Detecting AI-Generated Banglish Text with Explainability and Robustness Analysis

Machine-generated-text detection has expanded across languages, domains and generators, but Romanized code-mixed varieties remain weakly represented. We construct BanglishAIGT v1, an 8,100-text experimental benchmark of Banglish (Romanized Bengali mixed with English) containing 3,067 human e-commerce reviews and 5,033 direct generations from three commercial LLMs. We compare four word/character TF-IDF classifiers with mBERT and XLM-R on fixed generator-stratified splits. Clean-test performance is near ceiling: the validation-selected Linear SVM reaches 0.988 macro-F1 and XLM-R 0.991, and the two are statistically indistinguishable in a paired analysis. Removing capitalization, punctuation and emoji at test time costs Linear SVM 1.3 macro-F1 points but costs mBERT 17.6 and XLM-R 14.8. Removing AI e-commerce examples from training lowers Linear SVM macro-F1 on e-commerce text from 0.978 to 0.864, whereas all six models retrained within the e-commerce domain alone still reach 0.960–0.983. For mBERT, SHAP attributions to commas and periods on 250 randomly sampled texts remain positive under resampling, although the wider token ranking is unstable. Together, these results give converging, non-causal evidence that near-perfect matched-split accuracy partly reflects formatting, domain and generator cues. We present the benchmark as a case study in why accuracy should be reported together with uncertainty, perturbation, restricted training and attribution, not as a detector ready for deployment.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24273351
Primary Topic
Authorship Attribution and Profiling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

BanglishAIGT: Detecting AI-Generated Banglish Text with Explainability and Robustness Analysis

Najmun Nahar Hira, Mahir Al Shahriar
2 links
Zenodo (CERN European Organization for Nuclear Research)
Authorship Attribution and Profiling
preprint

BanglishAIGT: Detecting AI-Generated Banglish Text with Explainability and Robustness Analysis

Najmun Nahar Hira, Mahir Al Shahriar
preprint en

Abstract

Machine-generated-text detection has expanded across languages, domains and generators, but Romanized code-mixed varieties remain weakly represented. We construct BanglishAIGT v1, an 8,100-text experimental benchmark of Banglish (Romanized Bengali mixed with English) containing 3,067 human e-commerce reviews and 5,033 direct generations from three commercial LLMs. We compare four word/character TF-IDF classifiers with mBERT and XLM-R on fixed generator-stratified splits. Clean-test performance is near ceiling: the validation-selected Linear SVM reaches 0.988 macro-F1 and XLM-R 0.991, and the two are statistically indistinguishable in a paired analysis. Removing capitalization, punctuation and emoji at test time costs Linear SVM 1.3 macro-F1 points but costs mBERT 17.6 and XLM-R 14.8. Removing AI e-commerce examples from training lowers Linear SVM macro-F1 on e-commerce text from 0.978 to 0.864, whereas all six models retrained within the e-commerce domain alone still reach 0.960–0.983. For mBERT, SHAP attributions to commas and periods on 250 randomly sampled texts remain positive under resampling, although the wider token ranking is unstable. Together, these results give converging, non-causal evidence that near-perfect matched-split accuracy partly reflects formatting, domain and generator cues. We present the benchmark as a case study in why accuracy should be reported together with uncertainty, perturbation, restricted training and attribution, not as a detector ready for deployment.

Zenodo (CERN European Organization for Nuclear Research)
Shahjalal University of Science and Technology (BD)
Authorship Attribution and Profiling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon