Quality-Based rPPG Compensation With Temporal Difference Transformer for Camera-Based Driver Monitoring

Remote photoplethysmography (rPPG) is a method for monitoring pulse signal by utilizing a camera sensor to capture a facial video including variations in blood flow beneath the skin. Recently, rPPG advancements have enabled the measurement of an individual’s heart rate with a Root Mean Square Error (RMSE) of approximately 1.0 in controlled indoor environments. However, when applied in car dataset including driving environments, the RMSE of rPPG measurements significantly increases to over 9.07. This limitation, caused by motion-related artifacts and fluctuations in ambient illumination, becomes particularly noticeable while driving, resulting in a Percentage of Time that Error is less than 6 beats per minute (PTE6) of up to 65.1%. To address these limitations, we focus on the assessment of rPPG noise, with an emphasis on evaluating noise components within facial video and quantifying quality of the rPPG measurement. In this paper, we propose a deep learning framework that infers rPPG signal and quality based on video vision transformer. the proposed method demonstrates that the top 10% quality measurements yield PTE6 of 91.98% and 99.59% in driving and garage environments, respectively. Additionally, we introduce a quality-based rPPG compensation method that improves accuracy in driving environments by predicting rPPG quality based on noise assessment. This compensation method demonstrates superior accuracy compared to the current state-of-the-art, achieving a PTE6 of 68.24% in driving scenarios.