Trustworthy confidence in Kazakh POS tagging: calibration and selective decisions in a low-resource text-correction pipeline

Confidence is often treated as a secondary output of a POS tagger. In a deployed text-correction system, however, confidence becomes part of the decision-making process because fixed thresholds determine whether a prediction is handled automatically or referred for review. We examine the reliability of this signal for Kazakh using UD_Kazakh-KTB (1,078 sentences, 10,536 syntactic words). Three transformer taggers, KazBERT, KazRoBERTa, and XLM-R, are compared with a feature-based CRF—the correction pipeline’s two-stage tagger—retrained under the same data conditions. All models use the same split; temperature is fitted on development data, and neural results are reported over three seeds. XLM-R is the strongest model (92.83% accuracy, raw ECE 5.45%), followed by KazBERT (91.05%, 6.43%) and KazRoBERTa (90.90%, 6.51%). The CRF reaches 83.31% accuracy and is sharply over-confident (mean confidence 0.97, raw ECE 13.66%). Temperature scaling reduces transformer ECE by 57–67% without changing predicted labels and, in a sensitivity analysis, reduces CRF ECE by 71%. At the 0.85 threshold, the raw CRF’s selective risk is 13.91%, compared to 5.5–6.5% for the raw transformers. After calibration, transformer risk falls to 3.9–5.1%, as 77–131 predictions per model move below the threshold; their error rate is 31–33%, compared to 4–5% for those remaining above it. Because a scalar temperature leaves confidence ranking essentially unchanged, this gain reflects fewer predictions clearing a fixed threshold, not better selection. These findings do not show that calibration improves the final correction output. Rather, they demonstrate that recalibrating confidence alone can alter the routing of identical POS predictions.