Comparative Study on the Generalization Ability of Machine Learning and Deep Learning Algorithms for Quality Assessment of Wearable PPG Recordings

One of the major challenges in using photoplethysmography (PPG) sensors for heart rate monitoring in real-world settings is ensuring signal quality. This work evaluates and compares machine learning (ML) and deep learning (DL) methods for PPG quality assessment on a unique and comprehensive framework that includes different sensors, wavelengths, body measurement locations, and recording environments. The PPG signals from one proprietary and five publicly available datasets were labeled in terms of quality by comparing the PPG-derived heart rate to a reference heart rate estimated from simultaneous electrocardiograms. Diverse techniques based on common ML classifiers and one- and two-dimensional convolutional neural networks (CNN) were trained on a dataset and tested on the remaining ones. The results showed that several generated models performed comparably to previous studies when they were tested on datasets with similar measurement positions and sensors to the training database. Specifically, reductions in sensitivity, specificity, and F<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub>-score of less than 3% from training to testing were observed on some methods. Contrarily, they reported a notably poorer performance when tested on datasets presenting conditions different from the training. Even the best-performing model, based on the well-known, pre-trained CNN AlexNet, experienced a performance drop of over 20% in that situation. These findings show that the analyzed ML and DL methods lack the ability to generalize across PPG signals captured from diverse environments, sensors, wavelengths, and measurement locations.