Benchmarking open-source large language models on Portuguese Revalida multiple-choice questions
10 out of the 31 LLMs attained better than human level of performance in the Revalida benchmark, with 9 failing to provide coherent answers to the task. Larger models exhibited superior performance overall. However, certain medium-sized LLMs surpassed the performance of some of the larger LLMs.
Authors
- Gustavo Lenci Marques (ORCID: https://orcid.org/0000-0002-6057-0350)
- Rosário Veiga (ORCID: https://orcid.org/0000-0002-7135-8603)
- Murilo Guedes (ORCID: https://orcid.org/0000-0002-3495-852X)
- João Victor Bruneti Severino (ORCID: https://orcid.org/0000-0002-8649-6494)
- Pedro Angelo Basei de Paula
- Matheus Nespolo Berger
- Solano Amadori Todeschini (ORCID: https://orcid.org/0000-0003-3738-7563)
- Eduardo Augusto Roeder (ORCID: https://orcid.org/0000-0002-1016-3319)
- Filipe Silveira Loures (ORCID: https://orcid.org/0000-0002-7574-8254)
Institutions
- Pontifícia Universidade Católica do Paraná (BR)
- Centro Universitário de Belo Horizonte (BR)
- Grupo Santa Casa de Belo Horizonte (BR)
- The Ohio State University (US)
- Universidade Federal do Paraná (BR)
Publication Details
- Journal
- BMJ Health & Care Informatics
- Published
- 2025-02-01
- DOI
- https://doi.org/10.1136/bmjhci-2024-101195
- Citations
- 6
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 3.93