Medical education research quality (MERSQ) checklist development: Are searches of BEME and non-BEME reviews standard?: A mixed method study

Even though there has been a lot of research in medical education, the quality of it has not increased similarly. This study aimed to provide a valid and reliable user-friendly tool for evaluating search strategies in medical education systematic reviews. This mixed study was conducted in 2019 to 2021, including 3 phases: systematic search, developing a medical education research quality (MERSQ) checklist, and evaluation of the search quality of best evidence in medical education collaboration (BEME) and non-BEME reviews. Three hundred nineteen items were retrieved from the systematic search of PubMed, Embase, Scopus, Psychinfo, ERIC, and Google Scholar. Following ensuring acceptable criteria, 30 items were included in comprehensiveness or reproducibility guarantees. The results showed that the instrument had an the intra-class correlation coefficient of 0.922 (P = .002), the reproducibility guarantee had 0.903 (P = .003), and the comprehensiveness guarantee had 0.926 (P = .006). We also calculated inter-rater reliability and internal consistency using Cronbach alpha of 0.827 (P < .001) and an instrument the intra-class correlation coefficient of 0.978. Using MERSQ, the overall search quality (41.75 vs 31.25, P = .009), reproducibility (22 vs 14.50, P = .004), and comprehensive score (18.75 vs 15.75, P = .880) of BEME studies were higher than non-BEME ones. Moreover, we found only 30% of studies completed searching documents. The search strategy query concerning the selection of synonym terms received the lowest score among studies. This study led to the development of a valid and reliable checklist for evaluating the search quality of medical education systematic reviews. Utilizing the MERSQ checklist, we found that BEME studies had higher quality than non-BEME ones, making the results from BEME studies more reliable.