Comparative Analysis of Human and Large Language Model Performance in Pharmacology Multiple-Choice Questions

In this article, we study the answers generated by a selection of Large Language Models to a set of Multiple Choice Questions in Pharmacology, and compare them to the answers provided by students, to understand which questions in this clinical domain are difficult for the models when compared to humans and why.We extract the internal logits to infer probability distributions and analyse the main features that determine the difficulty of questions using statistical methods.We also provide an extension to the FrenchMedMCQA dataset, with pairs of question-answers in pharmacology, enriched with student response rate, answer scoring, clinical topics, and annotations on question structure and semantics.