Subgroup performance and safety flags in pre‐operative machine learning: a reply

We are grateful to Gao [1] for their correspondence on our study [2], which raises two questions relevant to translating our modelling work into clinical practice. We agree that surgical scheduling and capacity planning differ between elective and emergency workloads and therefore report a sensitivity analysis including only elective surgery, for which model performance was maintained. We do not agree that surgical specialty-specific subgroup reporting is useful because our aim was to make modelling congruent with clinical workflows. Both Leeds and Bradford run ‘general’ nurse-led pre-operative assessment services that manage multiple surgical specialties, as is common across the NHS. Patients with specific requirements or at high risk are referred to specialist face-to-face anaesthetist-led clinics after an initial assessment, in accordance with national guidance [3]. Furthermore, model performance was maintained mostly, even when the procedure (and by implication the specialty) was masked from the model entirely. In the typical NHS setting, specialty-specific thresholds are thus not congruent with the business question. Settings where pathways are organised in specialty siloes may require specialty-specific models, but we cannot assume necessarily that those siloes will be the same as specialties coded in the NHS. On the issue of safety, we agree that the performance of any warning flag should be evaluated. The warning flag we refer to in the study is intended as a safety feature during live use. When triggered, no prediction is generated. During training, values never fell outside the training range. We conducted evaluation against the whole validation set in order not to overreport model performance. Technical testing of the warning flag functionality formed part of the NHS England evaluation to accredit the system for being allowed to interface with GP Connect but quantifying how often it triggers in live use would require prospective study. We apologise for any confusion caused by including the warning flag in the methods section dealing with model development, rather than in a separate description of the clinical system integration. Gao also refers to important downstream process implications of early risk stratification such as differential laboratory testing strategies for low and higher complexity patients. This is an important future development and can further help increase the impact of any efficiency improvements related to early stratification and/or process automation [4]. Furthermore, when laboratory values and diagnosis codes become available in real time, as the GP Connect roadmap describes, previous laboratory variables will become available as predictive variables and laboratory values could even become prediction targets to guide testing strategies. We agree with Gao that subgroup performance and the performance of safety features require specific evaluation and that prediction modelling should integrate with diverse systems including laboratory testing. However, this is all best suited to further prospective study.