Automatic Identification of Citation Distortions in Biomedical Literature: A Case Study

ABSTRACT Citations are central to the propagation of scientific information. Ensuring the accuracy of citations is essential to maintain the credibility of scientific knowledge. However, assessing citations is a significant challenge, especially at large scale. This study examines the utility of natural language processing (NLP) in identifying poor citation practices. Specifically, we replicate Greenberg's 2009 study on citation distortions in Alzheimer's research, which demonstrated how poor citation practices can contribute to the establishment of unsubstantiated claims as facts. We explored two approaches: one that utilizes large language models (LLMs), and another that relies on existing publicly available NLP tools and publication metadata. Our findings suggest that, among Greenberg's three types of citation distortion – citation bias, amplification, and invention – automated approaches are most effective at detecting amplification, with more limited success in automatically replicating Greenberg's results for the other citation distortion types. Further refinements to LLM pipelines are needed to better capture the subtleties of citation bias and invention in biomedical publications.