Cross-lingual consistency of large language models: a six-language ICD-11 coding benchmark and evaluation methodology

Does a large language model (LLM) give the same label to the same content when that content is written in a different language? This cross-lingual consistency is not the same thing as accuracy, yet it has rarely been measured for tasks with a fixed label set. We present a benchmark and an evaluation method for it, using ICD-11 coding as the test case, in which a model reads an official disease definition and returns a code. The resource pairs 300 ICD-11 entities from 23 chapters with their official World Health Organization (WHO) definitions in English, Spanish, French, Chinese, Arabic and Russian. Because these are official translations, what the benchmark measures is consistency across the official ICD-11 language editions rather than pure language invariance. We grade predictions at four code levels, check every predicted code against the ICD-11 release, and measure agreement across languages with chapter-level Krippendorff’s alpha, treating the languages as raters. Across six current models (10,800 calls), chapter accuracy ranged from 14.3% to 83.6%, or from 9.3% to 63.2% if codes that do not exist in ICD-11 are counted as wrong, and chapter-level alpha from 0.48 to 0.92. The two measures did not always move together. One model was fairly consistent (alpha = 0.71) but reached the correct chapter in only 24% of cases, and on the items where all six languages agreed it was wrong twice as often as right. A repeated-sampling control showed that the differences between languages are larger than run-to-run variation. Agreement across languages alone cannot show that a model is correct, and we argue that both properties should be reported, at the same granularity, for any multilingual system that chooses from a fixed label set. The item set, prompts, scoring code and per-call logs are released.