Knowledge Distillation for Single Pixel Imaging Classification

Single-pixel imaging (SPI) is a computational imaging technique that captures high-dimensional signals using a single photo-detector by projecting a series of optical coded elements (OCEs) onto a scene. Each pattern modulates the light interacting with the scene, and the single detector acquires the intensity. Currently, these patterns are jointly optimized with a neural network to perform a specific task using an end-to-end (E2E) approach. However, the performance of E2E models significantly degrades when applied to physical constraint scenarios, such as rigorous compression ratios (CR) in a single-pixel camera to reduce acquisition and processing time. Knowledge distillation (KD) seeks to transfer the knowledge acquired by a high-capacity and performing large model (teacher) to a constrained small model (student). To address the limitations of E2E under strict constraints, we propose a KD framework for SPI classification. In this work, knowledge is transferred from a pretrained, less constrained model (teacher) that guides a physical constraint SPI model (student). In SPI classification tasks, the teacher benefits from richer information and a better-conditioned sensing matrix. Therefore, the student network learns to imitate the behavior of the teacher's probability distribution over classes, leveraging valuable representations captured by the teacher. Experimental results demonstrate that the student model, with a CR difference of 20%, have similar performance or can even outperform the teacher model. Moreover, this configuration enables the student model to benefit from an inference time of approximately 25 ms faster than the teacher model.