The Invisible Infrastructure of Biomedical Research: An Exploratory Analysis of Software Mentions and Sentence-Level Citation Marking

Abstract Background Research depends on software, yet software is seldom cited formally. Existing prevalence estimates come mainly from small, hand-coded samples and are usually reported as a single aggregate rate, which cannot show whether under-citation differs between classes of tool. Methods We conducted an exploratory, descriptive analysis of a deterministically defined subcorpus of the Chan Zuckerberg Initiative (CZI) Software Mentions dataset (799,756 mentions in 138,019 biomedical papers, 1980–2022). Mentions were assigned to five mention styles and six software types using published regular-expression and lexicon rules. The outcome, citation marking, was defined as the presence of a citation marker in the sentence that contains the software name; it is distinct from reference-list citation. We used contingency analysis (Cramér’s V) and logistic regression with paper-clustered standard errors. A name-ambiguity diagnostic compared registry matching with a rule-based context classifier on 1,961 curated mentions of eight terms. The dataset’s curation labels were not used as a filter: 15.0% of definitively labelled mentions are classified as not software. Results Overall, 10.7% of mentions were citation-marked and 69.7% were bare in-text names. Marking differed by software type (Cramér’s V = 0.219), from 2.2% for statistical packages and 1.0% for reference managers (7 of 731 mentions) to 18.0% for machine-learning frameworks. Relative to domain-specific tools, the adjusted odds of marking were higher for machine-learning frameworks (OR = 1.70) and programming languages (OR = 1.47) and lower for statistical packages (OR = 0.23). Developer attribution was associated with lower odds of marking (OR = 0.29) and URLs with higher odds (OR = 1.28). The analysis found no robust time trend. The registry contained none of the eight ambiguous terms, and the context rules reached F1 = 0.591, a transparent baseline well below current transformer-based systems. Conclusions In this subcorpus, sentence-level citation marking of software is low and differs systematically by type of tool, being lowest for general-purpose statistical software. Because the instruments lack validation against human annotation and the subcorpus is not a probability sample, the findings are exploratory; they indicate where software-citation policy and validation work could most usefully be focused.