Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID‐19
From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.
Authors
- Joe Packham
- George Nicholson (ORCID: https://orcid.org/0000-0001-9588-6075)
- Ana Tendero‐Cañadas
- Vasiliki Koutra (ORCID: https://orcid.org/0000-0002-1117-4554)
- Ivan Kiskin (ORCID: https://orcid.org/0000-0002-2551-840X)
- Björn W. Schuller (ORCID: https://orcid.org/0000-0002-6478-8699)
- Steven G. Gilmour
- Stephen Roberts (ORCID: https://orcid.org/0000-0002-9305-9268)
- Richard Payne (ORCID: https://orcid.org/0000-0002-1721-3868)
- Alexander Titcomb
- Davide Pigoli (ORCID: https://orcid.org/0000-0003-4591-4167)
- Jonathon Mellor
- Kieran Baker (ORCID: https://orcid.org/0000-0002-3312-4806)
- Radka Jersakova
- Lorraine Butler
- David J. Hurley (ORCID: https://orcid.org/0000-0003-2108-4496)
- Sabrina Egglestone
- Tracey Thornley (ORCID: https://orcid.org/0000-0002-9276-052X)
- Harry Coppock (ORCID: https://orcid.org/0000-0003-2404-8319)
- Jobie Budd (ORCID: https://orcid.org/0000-0003-3337-6859)
- Selina Patel (ORCID: https://orcid.org/0000-0001-7309-1917)
- Chris Holmes (ORCID: https://orcid.org/0000-0002-6667-4943)
Institutions
- University of Nottingham (GB)
- King's College London (GB)
- Centre for Life (GB)
- University of Surrey (GB)
- University of Oxford (GB)
- King's College School (GB)
- The Alan Turing Institute (GB)
- Science Oxford (GB)
- UK Health Security Agency (GB)
- University College London (GB)
- Imperial College London (GB)
- University of Brighton (GB)
Publication Details
- Journal
- Statistics in Medicine
- Published
- 2024-09-05
- DOI
- https://doi.org/10.1002/sim.10211
- Primary Topic
- COVID-19 diagnosis using AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Royal Statistical Society
- Alan Turing Institute
- National Institute for Health and Care Research
- Department of Health and Social Care
- NIHR Oxford Biomedical Research Centre