Which Distortion gets confused? A Controlled Comparison of Language Models for Dominant Cognitive Distortion Classification

Cognitive Distortions are unhelpful thinkingpatterns that Cognitive Behavioral Therapy(CBT) aims to identify and restructure, soclassifying cognitive distortions is a crucialprocess for reframing. The classificationbecomes difficult as the data is limited, andthe data are imbalanced most of the time. Westudy dominant distortion classification onthe publicly available Therapist Q&A dataset(Shreevastava and Foltz, 2021) where the anno-tator agreement on the dominant distortion islow (34%). Using the annotated distorted spanas input, we compare TF-IDF baselines withRoBERTa, DeBERTa-v3 and MentalBERT(with and without focal loss) over three seeds,reporting accuracy, macro F1 and Cohen’skappa with significance tests. The encodersreach 0.39 to 0.44 accuracy, well above themajority-class baseline (0.15), the three cross-entropy encoders are not significantly differentfrom each other, while the weighted focal lossis worse. An error analysis shows that errorsconcentrate on a small group of overlappingdistortions, mainly All-or-nothing thinking,Magnification and Mental filter, whichsuggests that unclear category boundaries limitperformance more than model choice.