Leveraging Large Language Models for Zero-Shot Classification of Psychiatric Diseases

In Natural Language Processing (NLP), large language models (LLMs) have demonstrated notable success, particularly in evaluating their performance on text classification tasks related to psychiatric diseases. This study employs the LLM to classify disease names based on users' queries by leveraging a dataset consisting of 10,000 health-related questions with their corresponding answers. Zero-shot prompting was used for classification since the models were not fine-tuned on medical datasets before predicting diseases. Three language models were tested: Mistral- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$8 \times 7 ~\mathrm{B}$</tex>, Llama- 8B, and Gemma-7B. Mistral- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$8 \times 7 ~\mathrm{B}$</tex> achieved the highest accuracy at 93.5 % and the best prompt adherence at 97.1 %. It was followed by Llama8B, which obtained an accuracy of 85.3 %. Gemma-7B recorded the lowest accuracy at 80.2%. The models were implemented using the ChatGroq API, and their performance was evaluated based on accuracy, adherence to instructions, and the capacity to distinguish between similar diseases. The structured prompt was created to retrieve only the main name of the disease, ignoring all irrelevant information and subtypes. It appears that Mistral <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$8 \times 7 ~\mathrm{B}$</tex> performed better at distinguishing between very similar diseases, while Llama-8B occasionally confused closely related conditions. On the other hand, Gemma-7B had precision issues but excelled in conversational scenarios. The results indicate that zero-shot prompting with LLMs can be effectively applied to disease classification, regardless of whether the models have been fine-tuned for this specific domain.