Semantic Search and Retrieval-Augmented Generation for Academic Literature Analysis Using ColBERT

Since the research field is expanding so quickly, a reliable approach for doing a literature review is needed. Various techniques and tactics have been proposed for this, and they have been used excessively. However, there are various disadvantages; thus, to overcome these shortcomings, the proposed system uses retrieval-augmented generation (RAG) pipelines alongside with robust vector embedding models to offer quick, contextually relevant answers and accurate responses to queries related to academic research. The proposed method starts with data set preparation, which integrates metadata such as titles, abstracts, and categories from academic data sets, like arXiv. In the embedding process, the HuggingFace model is used to generate semantic vector representations of the data set. After that, these embeddings are stored in a vector database using ChromaDB, which facilitates scalable and persistent retrieval based on vector similarity. In the proposed method, a configurable query engine is used that retrieves the top results by using parameters like similarity_top_[Formula: see text] and then enhances the relevance through post-processors ColBERT. After this, the system presents a curated collection of pertinent research papers on specific topics such as knowledge graphs, video generation using diffusion models, and machine learning that are given by the user as a query for the method. The system achieves average user satisfaction ratings of 4.6–4.8/5.0 and relevance scores of 0.65–0.76, showing specific improvements over traditional search methods. The precision, recall, and [Formula: see text]-scores are enhanced by 15–20% as compared to baseline retrieval due to post-processing.