Integrating Vector Stores and Information Retrieval for Analysis Literature Data on Mycobacterium tuberculosis: experience feedback

Tuberculosis, caused by Mycobacterium tuberculosis, remains a significant public health issue globally, exacerbated by the growing prevalence of multidrug-resistant strains and the lack of new therapeutic options. Despite extensive research efforts, there remains a critical need to effectively harness the vast amount of available data to drive new insights and treatment strategies. Currently, over 200,000 genomes of Mycobacterium tuberculosis are publicly available, and there are more than 100,000 scientific articles on PubMed concerning this bacterium. The potential for breakthroughs is immense if we can systematically study these two corpora in conjunction. However, the scale of these datasets necessitates the development of new tools based on big data analytics, artificial intelligence, and large language models (LLMs). This paper presents a study on the application of information retrieval (IR) techniques for performing Retrieval-Augmented Generation (RAG) to facilitate access to current knowledge about Mycobacterium tuberculosis. The approach integrates IR and response generation to provide relevant and contextually appropriate answers to user queries, leveraging scientific documents and genomic data.