Automating Theory of Mind Assessment with a LLaMA-3-Powered Chatbot: Enhancing Faux Pas Detection in Autism

The Theory of Mind (ToM) is critical for social interactions, enabling individuals to understand and predict others' mental states. Deficits in ToM, particularly in detecting faux pas, are common among individuals with Autism Spectrum Disorder (ASD). Traditional methods for assessing ToM, such as the Faux Pas Recognition Test (FPRT), require significant human involvement, which can limit accessibility and scalability. This study aims to explore how large language model (LLM)-powered chatbots, specifically utilizing the LLaMA-3 model, can enhance the assessment of ToM by automating and improving the administration of the FPRT. The methodology involved creating a dual-phase process where the chatbot first assesses user responses and then provides rehabilitative guidance to correct misunderstandings. Additionally, the system employs a scoring mechanism combining automated evaluation with human oversight to ensure accuracy and consistency, with system prompts crafted based on chain-of-thought prompting to enhance the depth and reliability of the assessments. Our findings demonstrate the effectiveness of LLM-powered chatbots in delivering dynamic and supportive ToM assessments, suggesting its potential for broader application for both artificial intelligence and psychological practices.