Performance Benchmarking of Open-Source Large Language Models on the Brazilian Society of Cardiology's Certification Exam

Background: Large language models (LLMs) have made a significant impact in medicine and demonstrate substantial promise for further development. However, most of the existing research has predominantly centered on English-language tasks with lower medical complexity. This underscores the importance of investigating the performance of state-of-the-art LLMs in more complex specialties, such as cardiology, and in languages beyond English, such as Portuguese. Objective: This study aimed to evaluate and compare leading LLMs based on their performance on the validated cardiology [...]