Predicting The Scholarly Impact of Research Papers Using Retrieval-Augmented LLMs

Assessing a research paper's scholarly impact is an important phase in the scientific research process; however, metrics typically take some time after publication to accurately capture the impact.Our study examines how Large Language Models (LLMs) can predict scholarly impact accurately.We utilize Retrieval-Augmented Generation (RAG) to examine the degree to which the LLM performance improves compared to zero-shot prompting.Results show that LLama3-8b with RAG achieved the best overall performance, while Gemma-7b benefited the most from RAG, exhibiting the most significant reduction in Mean Absolute Error (MAE).Our findings suggest that retrieval-augmented LLMs offer a promising approach for early research evaluation.Our code and dataset for this project are publicly available 1 2 .