Automated Working Alliance Assessment in Psychological Counseling Using Gemini and XGBoost

Session dialogue assessment based on machine learning is gradually becoming an effective solution for therapeutic alliance measurement which is an important factor for successful psychotherapy. However, most existing models assume clean and pre-structured dialogue transcripts, whereas real-world counseling documentation often contains heterogeneous case reports. This gap limits the applicability of current automated assessment models in realistic documentation scenarios. In this work, we propose a framework for automated working alliance assessment from complex, multilingual reports. First, language-specific BERT models are fine-tuned to process case reports across different languages, enabling accurate speaker role delineation and dialogue structuring. Second, Gemini-2.5-Flash is leveraged to annotate the dialogues with working alliance ratings. Third, a hybrid feature representation strategy is then developed to jointly capture linguistic style and semantic content from the counseling dialogues. Furthermore, an entropy-based mutual information analysis is conducted to identify the most informative linguistic features. Finally, the extracted hybrid features serve as inputs to XGBoost for alliance assessment. In experiments, the proposed framework shows better performance in the comparison with SOTA methods and generalization ability.