An Enhanced Reference Paper Collection System with Generative AI for Paper Relevance Assessment

Nowadays, academic paper search systems help researchers find related studies from a large number of publications. However, assessing whether each paper matches the user search query remains very difficult. Previously, we proposed a reference paper collection system, which assesses paper relevance using sentence embedding similarity. Unfortunately, the results are not always agree with human expert judgments because the system relies only on semantic similarity, whereas human experts considered predefined relevance criteria when evaluating papers. In this paper, we propose an enhanced reference paper collection system by integrating Generative Artificial Intelligence (GenAI) into the relevance assessment process. Candidate papers are retrieved from multiple academic sources and selected through a two-stage retrieval process before being assessed by the Mistral-Small-24B-Instruct-2501, a Large Language Model (LLM), using their titles, abstracts, disciplines, and keywords. Using predefined relevance criteria derived from human expert evaluations and a few-shot prompting approach, the LLM assigns one of four relevance categories and generates a short explanation for each assessment. The enhanced system was evaluated using 20 queries from five academic fields with human expert annotations as the reference standard. Compared with the results of the previous system, it improved relevance assessment agreement by 31.77% in Gwet’s AC2 and paper ranking quality by 11.27% in NDCG. In addition, the LLM was instructed to generate explanations based on the provided paper metadata to reduce unsupported statements. The generated explanations were evaluated using the RAGAS Faithfulness metric (0–1 scale) and an AspectCritic-based explanation relevance metric (1–3 scale), achieving average scores of 0.701 and 2.73 respectively, indicating that the generated explanations were generally supported by the provided paper metadata and appropriately described the relationship between the user query and retrieved papers. The average inference cost was approximately $0.000277 for each paper relevance assessment request. This enhanced system shows promising results in providing relevance assessments with explanations in our experiment across five academic fields.