Image-free single-pixel sensing for human pose estimation and parameter-efficient fine-tuning

Current mainstream methods for human pose estimation (HPE) rely on high-resolution images or videos, which suffer from high hardware costs and privacy leakage risks. Additionally, existing single-pixel sensing methods in this field exhibit insufficient accuracy and poor generalization. In this Letter, we propose a novel, to the best of our knowledge, image-free single-pixel sensing for HPE technique that encodes scenes into a small number of measurements, recovers intermediate feature representations from them, and then directly performs efficient and robust two-dimensional (2D) and three-dimensional (3D) HPE. Experiments show that at a sampling rate of 6.25%, the proposed technique achieves 67.9 AP for 2D pose estimation on the COCO dataset, which significantly outperforms models with previous state-of-the-art CNN backbones, and achieves 3D pose estimation on the Human3.6M dataset that also outperforms the baseline model SimpleBaseline3D. Meanwhile, we introduce the parallel Adapter into the Feed-Forward Network (FFN) of the Vision Transformer-Huge (ViT-H) backbone for fine-tuning. At a sampling rate of 9.375%, only 45.1% of the parameters need to be trained, which outperforms the full-parameter trained model. This technique is expected to reduce training computational and hardware costs, protect privacy, and achieve “image-free single-pixel sensing+" multi-task generalization.