UsmleGPT: An AI application for developing MCQs via multi-agent system

Driven by the trending multi-agent system (MAS) that harnesses collective intelligence from numerous large language models (LLMs), UsmleGPT is a Python-based application designed to enhance LLM-generated content tailored for the USMLE Step 1 scenario. The MAS aligns with the NBME’s framework, incorporating advanced prompt strategies to guide each LLM through their respective tasks and specialties. This enables UsmleGPT to surpass conventional practices in automating item generation. Beyond the original script, a freely accessible, user-friendly website complements the tool, facilitating its adoption. UsmleGPT represents a breakthrough in medical education, transforming medical exam preparation and setting new standards for high-stakes medical assessments. • Novel multi-agent system (MAS) integrates LLMs for USMLE Step 1 item generation. • Outperforms mainstream LLM approaches in quality, diversity, and relevance of generated items. • Aligns with NBME standards, enhancing quality and efficiency of test development. • Utilizes CoT-SC and ToT strategies to improve LLM reliability and problem-solving. • Addresses item writer deficits, offering a scalable solution for high-stakes exams.