In Reply: Arms Race Control Score Standardizes Residency Applicant Publication Assessment

To the Editor: The authors would like to thank Pulido et al1 for engaging with our article and sharing their unique perspective. The authors raise numerous points related to the Arms Race Control Score (ARCS) methodology and potential ARCS implications for the “publication arms race.”2 First, Pulido et al critique the score in stating that “high effort” research is not clarified, as it may relate to “time investment, research complexity, or field impact,” which may introduce bias and potential subjectivity in scoring. However, ARCS' utility lies in its ability to provide an objective, unbiased scoring of research effort irrespective of institution. As the very name of the Publication Effort Score (PES) demonstrates, the number of points from 1 (editorials/historical/commentaries/letters) to 4 (clinical/basic science/meta-analysis/cadaver papers) is grounded in how much effort is typically required to produce each publication type. Journal impact factor is also considered, as publications in a top 500 journal (impact factor >10.2 at time of publication) receive an additional 1 point to their PES. The publication value unit is directly calculated from PES, and ARCS only includes publications with publication value unit >1, thus prioritizing higher effort scholarly works. ARCS may be paired with our qualitative research assessment pyramid (Figure 5) to assess an applicant's research capabilities holistically. Next, Pulido et al suggest that ARCS encourages applicants to produce “‘high effort' studies rather than fostering impactful research.” Neurosurgery residency programs receive hundreds of applications each year, with most faculty typically unable to provide a thorough review of every applicant's curriculum vitae. The anonymous neurosurgery residency program in our study interviewed 57 applicants with 537 cumulative publications, representing only a small proportion of total applicants. Previous research has emphasized the exponential growth of ERAS-eligible “research items” listed by neurosurgery applicants. Pulido et al suggest that ARCS may encourage “gaming” the system, with applicants pursuing clinical studies over reviews and commentaries. However, that is precisely a primary objective of ARCS: Higher effort publications should be incentivized over lower effort publications. The production of new evidence (clinical articles; PES = 4) should be rewarded over the synthesis of existing evidence (systematic reviews; PES = 3). In the current system—which is already being “gamed”—10 commentaries as a fourth author and 10 clinical studies as a first author both qualify as 10 publications, despite significant variation in overall authorship effort. We acknowledge the importance of academic discourse through letters and commentaries, and multi-institution research serves as a foundation for many of our most impactful studies. Nevertheless, we recently demonstrated the apparent laxity by which authorship is awarded—the percentage of publications in Neurosurgery with 5 authors or less declined from 91% in 1977 to 37% in 2023.3 Anecdotal evidence notes that some authors may pursue letters, commentaries, or reciprocal “gift authorship” to colleagues in their network to inflate their CV with “easy publications.” ARCS will properly differentiate these applicants, offering applicant evaluators an automated, objectively comparable score to use. Figure 4 in our manuscript illustrates this concept. We commend Pulido et al1 for commenting on disparities in access to research opportunities, which we highlighted in “The Current Residency Applicant Publishing Ecosystem” paragraph. We described the flawed current system based on total publications, which does not account for the clear inequality in access to research opportunities among students of different backgrounds. Medical students with a home neurosurgery residency training program have greater research access, more publications, less evaluative data (ie, grades, distinction), and a significantly increased probability of matching. Pulido et al rightfully note that disadvantaged students have barriers in accessing high-impact research, and they believe that this trend may be exacerbated by ARCS. They suggest that citation counts, h-index, or journal impact factor may provide a more accurate assessment of an applicant's research contributions. However, ARCS seeks to standardize high-effort research on behalf of the author. The mentioned alternatives are significantly flawed for students—h-index favors career duration, and total citation count is a function of the publication regardless of contributions, which likely favors students from elite institutions who get their name on high-impact basic science research. The Stop The Arms Race study was developed to address the grave concern about disparities in research opportunities, specifically by developing baseline ARCS metrics across applicant types to allow meaningful comparisons of scholarly work. Our goal was for ARCS to be an objective tool that improves how we measure residency applicant research productivity, as there have been numerous calls in the literature for an objective metric to aid in resident selection. ARCS is by no means a one-size-fits-all solution to the current state of student academic publishing. Producing studies that use the highest level of evidence and publishing them in leading journals remains a noble objective for almost all physician-scientists. ARCS may not be appropriate for students in some fields, including engineering, technology, or mathematics where fewer, but higher impact, publications are the focus. Moreover, we do not suggest that ARCS be the determining factor of an applicant's suitability for neurosurgery residency. In fact, if ARCS achieves our goal, it can slow the frantic pace of medical student publishing and allow applicants to shift their efforts to other initiatives that round out a holistic application.