Comparative analysis of AI chatbot mathematical performance

Abstract In this research, we assessed four publicly available large-language models(LLM): ChatGPT-4o, Claude 3.5 Sonnet, DeepSeek V2, and Gemini 1.5 Pro, to study a curated set of 10 math questions, varying in fields such as algebra, arithmetic, trigonometry, geometry, and statistics, categorised by topic and different levels of complexity. We used Text analysis, Cosine similarity, Jaccard similarity and a Column chart of word-count to compare the results given by each model. To analyse the performance thoroughly, we applied Analysis Of Variance (ANOVA) for word count, Fleschscore, Polarity and Sensitivity analysis on the results to test accuracy, clarity of explanation, range of problems supported, ease of platform navigation and possible errors made by the chatbots. Our findings show slight variation in the performance among the chatbots. This research also highlights the limitations of each model providing insights to make optimum use of these chatbots in solving mathematical problems.