Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Importance: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. Objective: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. Design, Setting, and Participants: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. Main Outcomes and Measures: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. Results: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. Conclusions and Relevance: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

Authors

Institutions

Publication Details

Journal
UNC Libraries
Published
2024-12-17
DOI
https://doi.org/10.17615/h78b-3y92
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Mitsuhiko Yamada, Hong Jin Jeon, Juliet E. M. Nakku, Qing Zhi Zeng et al.
UNC Libraries
Machine Learning in Healthcare
article

Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Mitsuhiko Yamada, Hong Jin Jeon, Juliet E. M. Nakku, Qing Zhi Zeng, Kim M. Kiely, Jane Fisher, Federico M. Daray, Crick Lund, Aaron Conway, Philippe Persoons, Kerrie Clover, Laura Marsh, Bizu Gelaye, Bruce Arroll, Brian P. Marx, Lesley Stafford, Peter Butterworth, Zelalem Negeri, Scott B. Patten, Anna Beraldi, Ankur Krishnan, Pim J. Cuijpers, Alexander W. Levis, Çharles N. Bernstein, Adam Simning, Keiko Suzuki, Brian W. Pence, Anthony McGuire, Sultan H. Alamri, Leanne Hides, Gregory Carter, Sujit D. Rathod, Flávia de Lima Osório, Thach Tran, Brooke Levis, Simone Honikman, Tiago N. Munhoz, Eric Green, Katrin Reuter, Leila Gholizadeh, María Iglesias-González, Charles H. Bombardier, Laura Navarrete, Christina M. van der Feltz‐Cornelis, Daphna Harel, Miranda T. Schram, Ryna Imma Buji, Sarah Markham, Catherine G. Greeno, Brian J. Hall, Simon M. Gilbody, Martin Härter, Iná S. Santos, Kirsty Winkley, Liisa Hantsoo, Stevan E. Hobfoll, Pei Lin Lynnette Tan, Terence J. Quinn, Alasdair G Rooney, Mahrukh Imran, Hamid Reza Baradaran, Ma. Asunción Lara, Yin Wu, Angelo Picardi, Holly Frances Levin-Aspenson, Eileen H. Shinn, Inge Petersen, Abbey Sidebottom, Felix Fischer, Lena Spangenberg, Felicity Anne Goodyear-Smith, Jill T Boruff, Manote Lotrakul, Roy C. Ziegelstein, Marcos Hortes Nisihara Chagas, Thandi van Heyningen, Liat Ayalon, Elmārs Rancāns, Nathalie Jetté, Danielle B. Rice, Mohammad E. Khamseh, Sônia Regina Loureiro, Lorie A. Kloda, Shen‐Ing Liu, Sharon C. Sung, Nagendra P. Luitel, Lai Fong Chan, Jesse R. Fann, Juwita Shaaban, Alyna Turner, Masatoshi Inagaki, Brett D. Thombs, Sherina Mohd Sidik, Bernd Löwe, Lynne I. Wagner, Karen Wynter, Daniel S S Fung, Sally Field, Mary A. Whooley, Dixon Chibanda, Paul A. Vöhringer, Brandon A. Kohrt, Thomas Hyphantis, Arvin Bhana, Ruth Ann Marrie, Kira E. Riehm, Martin Taylor‐Rowan, Andrea Benedetti, Suiqiong Fan, Jennifer White, Yemisi Takwoingi, Juliana C.N. Chan, Yuying Zhang, Chen He, JianLi Wang, Ying Sun, Dipika Neupane, Dagmar Amtmann, Parash Mani Bhandari, Marleine Azar, John P A Ioannidis, David Watson, Yeates Conwell, Yunxin Kwan, Janneke M de Man-van Ginkel, Stephanie L. Pugh, Kumiko Muramatsu
article en

Abstract

Importance: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. Objective: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. Design, Setting, and Participants: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. Main Outcomes and Measures: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. Results: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. Conclusions and Relevance: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

UNC Libraries
University of North Carolina at Chapel Hill (US)
George Washington University, University of Washington, Johns Hopkins University, University of Pittsburgh, University of Rochester, New York University Shanghai, Harvard T.H. Chan School of Public Health, McGill University, Iran University of Medical Sciences, McGill University Health Centre, York University, King's College London, Australian National University, University of Technology Sydney, University of Wollongong, Monash University, Queensland University of Technology, University of Queensland, Vrije Universiteit Amsterdam, Sungkyunkwan University, Bar-Ilan University, King Abdulaziz University, Shimane University, Universiti Kebangsaan Malaysia, Inyuvesi Yakwazulu-Natali, Chinese University of Hong Kong, Universidad de Buenos Aires, Universidade de São Paulo, Humboldt-Universität zu Berlin, University of Cape Town, Freie Universität Berlin, Instituto Nacional de Psiquiatría Ramón de la Fuente Muñiz, Berlin Institute of Health, University of Ioannina, Samsung, Duke Global Health Institute, Duke University, Medical Center, University of Rochester, Canadian Institutes of Health Research, Universitätsklinikum Hamburg-Eppendorf
Openalex Percentile: Top 79%
Machine Learning in Healthcare
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.