Preliminary evaluation of a native Turkish large language model in emergency triage: a pilot study on cultural vs. standard terminology

Aim: Global Large Language Models (LLMs) often struggle with cultural nuances. This pilot study investigates the feasibility of “Kumru,” a developing native Turkish LLM, in interpreting culturally specific emergency triage vignettes compared to standard medical terminology.Materials and Methods: A prospective in-silico study was conducted using a pilot dataset of 50 clinical vignettes based on Turkish Ministry of Health (Annex-4) Guidelines. The dataset was stratified into standard medical terminology (n=31) and cultural idioms (n=19) to assess domain-specific linguistic processing.Results: The model achieved 93.7% sensitivity for high-acuity cases but exhibited a specificity of 0% and an over-triage rate of 52%. While overall diagnostic accuracy was moderate (44.0%), the model demonstrated statistically superior performance in decoding cultural idioms (84.2%) compared to standard terminology (19.3%, p < .001).Conclusions: Although the current native model demonstrates a high capability for understanding local idioms, its low specificity renders it operationally inefficient for autonomous clinical decision-making. These findings present a proof-of-concept for linguistic localization but highlight the necessity for extensive fine-tuning before clinical deployment.