Identifying <scp>COVID</scp> ‐19 Disease Severity in Real‐World Data: Implications for Medical Product Effectiveness Studies

ABSTRACT Purpose The U.S. Food and Drug Administration (FDA) defined disease severity criteria to assist clinical development of medical products for management of COVID‐19. These definitions were translated to code‐based algorithms for use in real‐world data. We validated the algorithms' performance in ambulatory settings at three regional integrated healthcare delivery systems contributing data to FDA's Sentinel System. Methods We identified cohorts of individuals ≥ 18 years that met the algorithms' criteria for mild, moderate, and severe COVID‐19 at incident COVID‐19 diagnosis or positive SARS‐CoV‐2 test, and separately, at incident COVID‐19 treatment, from January 2022 through April 2023. We validated the algorithms via chart review of a random sample to calculate positive predictive values (PPVs) and 95% CIs. Results The algorithms identified 33 071 patients at COVID‐19 diagnosis or positive test; 26 985 mild (49 chart reviewed), 5180 moderate (55 reviewed), and 906 severe (56 reviewed). A total of 4512 patients were identified at COVID‐19 treatment; 3474 mild (56 reviewed), 848 moderate (60 reviewed), and 190 severe (46 reviewed). The PPVs (1) at COVID‐19 diagnosis or positive test were: mild 57% (95% CI: 43%–71%), moderate 58% (95% CI: 45%–71%), and severe 54% (95% CI: 41%–67%), and (2) at COVID‐19 treatment: mild 57% (95% CI: 44%–70%), moderate 70% (95% CI: 58%–82%) and severe 72% (95% CI: 59%–85%). Conclusion The algorithms had low‐to‐moderate performance in classifying COVID‐19 severity in ambulatory settings, depending on assessment at diagnosis or treatment. Researchers should consider the performance of the algorithm when using real‐world data to assess COVID‐19 severity.