Leveraging Large Language Models to Extract Data Citations

Research data providers need to understand, measure, and elevate the impact of their re-search data. One prominent class of such data providers, dedicated research data centers (RDCs), are often situated within national statistical offices and central banks and are tasked with providing access to granular administrative data. Accurately measuring and tracing the usage of this data in research papers, however, remains a process which traditionally relies on human readers and is thus not only time consuming, but also prone to errors. Our contribu-tion aims to simplify and improve this process by exploring the potential of Large Language Models (LLMs), specifically GPT-3.5, to help automate the identification of data sources used in research papers. We aim to provide a solution for researchers and RDCs who need to identify unknown da-tasets from previously unseen research papers, particularly in the fields of economics and finance. Unlike prior literature that focuses on natural sciences or ML/NLP papers, our ap-proach targets standardized datasets used in economics and finance, which are often li-censed from commercial providers, public institutions, or individual researchers. Automated identification of these datasets is crucial for understanding their impact and how they are combined with other data sources. We propose two primary use cases for our dataset extraction pipeline: recommending data to researchers based on their interests and calculating data impact factors to justify investment in datasets. Our input sample includes all research papers from the Deutsche Bundesbank Discussion Paper series between February 1995 and July 2023, totaling 1,102 papers. We manually labeled a random sample of 103 papers to evaluate the performance of our ap-proach. Manual labeling involves human readers scanning papers for dataset mentions, re-cording synonyms, and noting the full text passages where datasets are discussed. This pro-cess highlights the complexity of dataset extraction due to the lack of standardized formats and sections within papers. Our methodology involves constructing a pipeline for assistant-style LLMs, specifically GPT-3.5, to automatically extract data citations from research papers. The pipeline consists of five steps: splitting papers into smaller segments, filtering out irrelevant segments, combining each segment with instructions for the LLM, querying the LLM for information, and consolidating the responses. We use various performance measures to evaluate the pipeline, focusing on recall to ensure we capture as many relevant dataset mentions as possible. Our findings suggest that LLMs hold considerable potential for automated data usage extrac-tion and downstream use cases which enable the work of RDCs: The model accurately iden-tifies dataset mentions verbatim in up to 62% of cases, and when considering synonymous dataset names, it achieves 100% accuracy. This demonstrates that the proposed method, outlined in detail within our contribution, can provide a valuable pathway for RDCs and data-providing institutions to understand the downstream effects of their data provision efforts and offer more efficient data provision services. Costs for using the pipeline are relatively low, ranging from USD 0.02 to USD 0.08 per paper. We recommend using larger context windows to reduce costs while maintaining performance.