When Language Rewrites History: Quantifying and Mitigating Cross-Lingual Inconsistency in LLM-Based Historical Question Answering via Quantitative Analysis and LLM Fine-Tuning
Open-ended historical questions can elicit different responses from a large language model when only the input language changes. We tested this with 200 questions about the Great Leap Forward and related modern Chinese history, written in English, Chinese, and Japanese. ChatGPT and Gemini generated 1,200 answers. We compared each answer with a reference in the same language using Emotion Risk (ER) and Embedding Distance (ED), combined these measures in a Final Score, and measured score variation across languages. Both models had their highest mean Final Scores for Japanese questions: 39.04 for ChatGPT and 39.68 for Gemini. We then split the questions by ID and fine-tuned Qwen2.5-1.5B-Instruct with supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). On 90 held-out language-level samples, mean Final Score fell from 34.55 in the original model to 34.36 after SFT and 31.88 after DPO. SFT gave the lowest mean Cross-Language Disparity (5.12, compared with 6.90 originally and 5.91 after DPO). These results show measurable language-conditioned differences in this historical question set and a reduction in the proposed response-risk score after training. The automatic metrics do not establish factual accuracy or historical bias on their own.
Authors
- Katelyn Chen
Institutions
- Carmel Research Center (US)
- Carmel Unified School District (US)
Publication Details
- Journal
- Applied and Computational Engineering
- Published
- 2026-11-17
- DOI
- https://doi.org/10.54254/2755-2721/2026.39098
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00