Exploring Dimensionality with Knowledge Distillation for Human Activity Recognition

Abstract Technologies that run on resource-constrained environments such as edge devices or locally hosted servers, led us towards a search for computationally lightweight models. In the present scenario, Human Activity Recognition (HAR) based on accelerometer and gyroscope sensors is one such domain where near real-time response is required and hence, we need to deploy the system to a local edge device. However, deep learning models that are often executed for activity prediction are not suitable for such resource constrained environment. To overcome this challenge, we have proposed an approach that combines the potential of data dimensionality and distillation of knowledge from a three-layer deep Convolutional Neural Network (CNN) model to train a single-layer CNN model. The novelty of the work lies in exploration of data dimensionality in a way that enables a student model to better generalize the knowledge captured by the teacher model. This proposed training pipeline allows us to implement simple models that attempt to mimic the performance of gigantic deep learning models without overfitting it. This methodology was tested on three different HAR benchmark datasets, UCI-HAR, WISDM and MHEALTH and they achieved overall classification accuracies of 96.78%, 90.01% and 99.12% respectively with more than 4 times less trainable parameter size.