Image Representation-Driven Knowledge Distillation for Improved Time-Series Interpretation on Wearable Sensor Data

With the increased demand for wearable sensors, image representations-such as persistence images and Gramian angular fields-transformed from time-series data have been investigated to address challenges in wearables arising from physiological variations, sensor noise, and limitations in capturing contextual information. To preserve the lightweight structural design of models, knowledge distillation (KD) has also been employed alongside image representations during training to distill smaller and more efficient models. Although image representations play a key role in providing richer and more informative features in training a model, their effectiveness within the KD framework has not been thoroughly explored. In this paper, we focus on image representation-driven KD to investigate whether these representations can provide useful knowledge leading to improved time-series interpretation in activity classification tasks. We explore the benefits of integrating image representations into KD, and we analyze the interplay between representation richness and model compactness with different combinations of teacher and student networks. We also introduce diverse KD strategies to utilize image representations, and we demonstrate the strategies with various perspectives, such as analysis of noises, generalizability, and compatibility, across datasets of varying scales to obtain comprehensive and insightful observations. These offer valuable insights for designing efficient and high-performance wearable sensor-based systems.