Artificial Intelligence (AI)-Assisted Coding for Competency-Based Skill Improvement: A Case Study of Motivational Interviewing Skill Development in Behavioral Health Providers-in-Training

Coding of session transcripts is a key strategy in assessing counselors’ and social workers’ fidelity to the motivational interviewing (MI) approach. Artificial intelligence (AI) shows promise in identifying patterns in session transcripts associated with MI proficiency. This study utilizes a multiple instrumental case study approach to compare and contrast intercoder reliability for Atlas.ti, versus human researchers, for six session transcripts using the Motivational Interviewing Competency Assessment (MICA). Atlas.ti’s scores and qualitative feedback were compared to human researchers, and interrater reliability calculated using the Intraclass Correlation Coefficient. Results indicate significant agreement between researchers and AI scoring of intention and strategy, as well as the composite score. The difference in reflection-to-question ratio scoring highlights a difference in AI’s ability to attend to the nuanced nature of utilizing MI skills and strategies to enhance client/patient motivation.