Reproducibility without vendor names: emphasizing design over branding in AI evaluation

To: Suzanne Bakken, PhD, RN, FAAN, FACMI, FIAHSI Editor-in-Chief Journal of the American Medical Informatics Association (JAMIA) Oxford University Press Dear Dr. Bakken, We thank the authors for their thoughtful correspondence and for raising important issues regarding transparency, reproducibility, and generalizability in evaluations of vendor-based artificial intelligence tools. We agree that these are important goals. However, in this instance, we do not believe that identifying the specific vendors evaluated in our randomized crossover trial would, by itself, substantially advance those goals. A central challenge in evaluating vendor-based AI products is that they evolve far more rapidly than most other clinical technologies or interventions. In the broader data science literature, authors can often specify model versions with some precision. In contrast, vendor-based products may change through frequent feature releases, workflow modifications, or institution-specific customization without a clear, standardized mechanism for communicating those changes to readers. Even during the course of our trial, both products continued to evolve. As a result, naming the vendors could create a false impression that any single vendor is persistently superior to another, which does not reflect how these systems are deployed in practice.