Unleashing the Power of Pretrained Transformer for Dense Prediction in Physiological Signals

The physiological signals obtained from advanced sensors, combined with deep learning techniques for classification and regression tasks, have become a core driving force in enhancing smart healthcare. Recently, dense prediction tasks for physiological signals-aimed at generating predictions that are closely aligned with the input signal to enable fine-grained analysis-have garnered increasing attention. The UNet family, often combined with sophisticated task-specific customizations, has become a popular choice to improve prediction performance. However, pretrained Transformers have recently revolutionized deep learning due to their powerful transferability and effectiveness. In this work, we aim to harness the power of pretrained Transformers for dense prediction, eliminating the need for extensive task-specific architecture design. We propose a simple yet universal encoder-decoder architecture that utilizes a pretrained Transformer encoder and a lightweight convolutional Restormer decoder for dense prediction on physiological signals. To optimize the trade-off between model performance and computational efficiency, we incorporate knowledge distillation (KD). Our experiments focus on four representative dense prediction tasks: blood pressure waveform (BPW) estimation, PPG-to-ECG (P2E) reconstruction, denoising, and fiducial point localization. The results show that our proposed architecture outperforms state-of-the-art models, validating the potential of pretrained Transformers in enhancing physiological signal processing and medical diagnostics. This approach marks a significant step forward in optimizing both the performance and efficiency of dense prediction tasks.