What can a national medical examination diagnose? A cross-sectional cognitive diagnostic modelling study from Brazil

Abstract Background Clinical competence is content-, context- and task-specific, yet national medical examinations usually report a single score. We used cognitive diagnostic modelling to compare attribute frameworks reflecting these views (clinical tasks, content areas, care settings and patient features) in the first edition of Brazil’s national examination of final-year medical students (ENAMED, 2025), and to examine what the raw score reveals about attribute mastery. Methods Cross-sectional psychometric analysis of public microdata from 39,256 final-year medical students who answered 90 multiple-choice items (two booklets aligned to a common item order). Ten Q-matrices representing four axes (clinical tasks, content areas, care settings and patient features, in original and revised versions) were fitted with the saturated G-DINA model. We compared absolute fit (RMSEA₂, SRMSR), relative fit (BIC) and classification accuracy; estimated attribute prevalence, mastery profiles and latent correlations; and used receiver operating characteristic (ROC) analysis to relate raw scores to full mastery. Results All specifications showed close absolute fit (RMSEA₂ 0.011–0.013; SRMSR 0.022–0.027), but the clinical-task Q-matrix had clearly the best relative fit (BIC 3,744,145 versus 3,755,939–3,767,002 for the other axes). The five tasks were distinct but positively correlated (r = 0.03–0.71); diagnostic-test reasoning was the least mastered task (35.4%) and the one missing in two-thirds of examinees who mastered exactly four tasks. Content areas behaved largely as a single clinical competence (inter-area r = 0.72–0.80), except mental health and public health, which formed a separate pair. Among care settings, only community and territory emerged as distinct. Patient age and sex did not define distinct competences, whereas sexual and reproductive health and social determinants of health formed largely independent dimensions. Revisions based on item rereadings improved relative fit. Raw scores identified full mastery accurately (area under the ROC curve 0.965–0.990; Youden-optimal cut scores 58–63 of 90) but could not indicate which attribute an examinee lacked. Conclusions In this national examination, a clinical-task framework yields the most informative diagnostic description of performance. Reporting attribute profiles by clinical task, alongside the overall result, could give students and medical schools actionable feedback, particularly on diagnostic-test reasoning.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11406696/v1
Primary Topic
Innovations in Medical Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

What can a national medical examination diagnose? A cross-sectional cognitive diagnostic modelling study from Brazil

Carlos Fernando Collares, Regiana Quinto de Souza, Luise Simonetti, Etyany Lays Mariano Costa et al.
Research Square
Innovations in Medical Education
preprint

What can a national medical examination diagnose? A cross-sectional cognitive diagnostic modelling study from Brazil

Carlos Fernando Collares, Regiana Quinto de Souza, Luise Simonetti, Etyany Lays Mariano Costa, Ana Luísa Oliveira Dornela, Giulia Gil Lopes
preprint en

Abstract

Abstract Background Clinical competence is content-, context- and task-specific, yet national medical examinations usually report a single score. We used cognitive diagnostic modelling to compare attribute frameworks reflecting these views (clinical tasks, content areas, care settings and patient features) in the first edition of Brazil’s national examination of final-year medical students (ENAMED, 2025), and to examine what the raw score reveals about attribute mastery. Methods Cross-sectional psychometric analysis of public microdata from 39,256 final-year medical students who answered 90 multiple-choice items (two booklets aligned to a common item order). Ten Q-matrices representing four axes (clinical tasks, content areas, care settings and patient features, in original and revised versions) were fitted with the saturated G-DINA model. We compared absolute fit (RMSEA₂, SRMSR), relative fit (BIC) and classification accuracy; estimated attribute prevalence, mastery profiles and latent correlations; and used receiver operating characteristic (ROC) analysis to relate raw scores to full mastery. Results All specifications showed close absolute fit (RMSEA₂ 0.011–0.013; SRMSR 0.022–0.027), but the clinical-task Q-matrix had clearly the best relative fit (BIC 3,744,145 versus 3,755,939–3,767,002 for the other axes). The five tasks were distinct but positively correlated (r = 0.03–0.71); diagnostic-test reasoning was the least mastered task (35.4%) and the one missing in two-thirds of examinees who mastered exactly four tasks. Content areas behaved largely as a single clinical competence (inter-area r = 0.72–0.80), except mental health and public health, which formed a separate pair. Among care settings, only community and territory emerged as distinct. Patient age and sex did not define distinct competences, whereas sexual and reproductive health and social determinants of health formed largely independent dimensions. Revisions based on item rereadings improved relative fit. Raw scores identified full mastery accurately (area under the ROC curve 0.965–0.990; Youden-optimal cut scores 58–63 of 90) but could not indicate which attribute an examinee lacked. Conclusions In this national examination, a clinical-task framework yields the most informative diagnostic description of performance. Reporting attribute profiles by clinical task, alongside the overall result, could give students and medical schools actionable feedback, particularly on diagnostic-test reasoning.

Research Square
Universidade Estadual de Santa Cruz (BR)
Good health and well-being, Quality education
Innovations in Medical Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon