An integrated set of citation network analytic approaches with qualitative methods to evaluate the impact of early articles in a body of literature over time: the case of intersex health

Recommendations for clinical practice from early, seminal publications may be widely cited and influential many years later, even when opposing evidence emerges. We employed a dynamic new combination of citation network analytic approaches with qualitative methods to measure and compare the impact of early articles in a body of literature over time. We apply these mixed-methods tools to the work of John Money, a controversial figure in intersex research. We 1) identified relevant peer-reviewed literature; 2) built a citation network; 3) measured impact through the citation counts and centrality metrics (via Katz Centrality and PageRank); 4) qualitatively classified citations of Money's work as positive, neutral, or negative and examined the valence of these citations over time. Our systematic search generated 103 articles; we built a citation network using the CrossRef API. Nineteen of twenty-nine Money articles (66 %) were among the 20 %-most- cited literature by citation count and centrality (p < 0.001). Attitudes toward Money's works shifted from mostly positive to mostly negative, with a low point in 2000, then became somewhat less negative starting around 2010 (p < 0.001 for valence differences). Our work provides a model using existing tools in a new way to illustrate how controversial work may remain widely cited for decades after it is called into question, though the valence of citations may become more negative over time. A full suite of tools that considers both citations (via network analyses) and the valence of citations (through qualitative approaches) provides a richer picture of the continuing influence on a body of work. • Despite previous controversies and questions about his research on individuals with variations in sex characteristics (VSC), John Money remains an influential figure in the field of intersex research. • Using an innovative combination of citation network-based analyses (via Katz Centrality and PageRank) and qualitative methods we assessed how much Money's early peer-reviewed publications influenced subsequent peer-reviewed citations. • Taken together, our systematic peer-reviewed literature search, citation network analysis, citation counts and centrality assessments, and qualitative classification of citations provide new insights into how early research continues to influence the empirical base of knowledge over time.