A Study of the Effects of Language on Large Language Model Responses and Human Perception

People often feel an idea more strongly in their mother tongue, and they soften a hard message so that the listener is not hurt. This study asks whether chat-based large language models (LLMs) handle hard messages with the same care in English, Hindi and Marathi. Ten industry situations, such as a rejected build, a loan denial and the removal of an engineer from duty, were given to ChatGPT, Claude and Gemini in the three languages. The 90 collected messages were scored with a concept-matched word list that has five harshness categories and four softening categories. The scores were Harshness H, Softening S and Net tone N = H − S, with content coverage and length as extra checks. Differences between models were large (Kendall’s W of 0.30 for N and 0.56 for S). Differences between languages were small and not significant after correction. Gemini wrote short messages with almost no softening (S = 0.06), whereas ChatGPT and Claude used courtesy, regret and offers of help. Gemini also left out more key facts in Marathi (82% coverage) than in English (94%). Each cell holds one response, the word list was built after reading the data, and no human rater was used, so the results are exploratory.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24360757
Primary Topic
AI in Service Interactions
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Study of the Effects of Language on Large Language Model Responses and Human Perception

Shubham Gunjal, Rupali Pawar, Siddhesh Ghodekar
Zenodo (CERN European Organization for Nuclear Research)
AI in Service Interactions
article

A Study of the Effects of Language on Large Language Model Responses and Human Perception

Shubham Gunjal, Rupali Pawar, Siddhesh Ghodekar
article en

Abstract

People often feel an idea more strongly in their mother tongue, and they soften a hard message so that the listener is not hurt. This study asks whether chat-based large language models (LLMs) handle hard messages with the same care in English, Hindi and Marathi. Ten industry situations, such as a rejected build, a loan denial and the removal of an engineer from duty, were given to ChatGPT, Claude and Gemini in the three languages. The 90 collected messages were scored with a concept-matched word list that has five harshness categories and four softening categories. The scores were Harshness H, Softening S and Net tone N = H − S, with content coverage and length as extra checks. Differences between models were large (Kendall’s W of 0.30 for N and 0.56 for S). Differences between languages were small and not significant after correction. Gemini wrote short messages with almost no softening (S = 0.06), whereas ChatGPT and Claude used courtesy, regret and offers of help. Gemini also left out more key facts in Marathi (82% coverage) than in English (94%). Each cell holds one response, the word list was built after reading the data, and no human rater was used, so the results are exploratory.

Zenodo (CERN European Organization for Nuclear Research)
AI in Service Interactions
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon