Integration of biomedical concepts for enhanced medical literature retrieval

Abstract This study investigates the potential of biomedical concepts—genes, diseases, and chemicals—in enhancing biomedical literature retrieval systems amidst the burgeoning volume of biomedical literature. We compare five traditional sparse approaches against hybrid BERT-based methods, to assess their effectiveness in leveraging biomedical concepts for improved retrieval accuracy. Our research poses critical questions on the necessity and efficiency of biomedical concepts in the era of advanced language models and evaluates if these concepts can further refine retrieval outcomes. Using datasets from Text REtrieval Conference Precision Medicine tracks (2017–2019), which are based on the Medline collection (30+ million biomedical publications) and NDCG@10 (Normalized Discounted Cumulative Gain) for evaluation, we demonstrate that biomedical concepts are indeed helpful for both hybrid and sparse retrieval strategies in the biomedical domain and that the performance of sparse and hybrid methods is comparable. We further demonstrate how our findings can be integrated into a live search system to support clinical practice.