A Lightweight Neural Network Model Based on Response-Based and Feature-Based Cross-Modal Knowledge Distillation for Image Classification Applications

This paper introduces a lightweight neural network model derived from a modified MobileNet architecture, optimized through response-based and feature-based cross-modal knowledge distillation. By working with Gray image and eliminating specific blocks from the original MobileNet, we significantly reduce the model’s parameters. The distillation process involves using MobileNet V1 trained on RGB data from the EDEN library dataset. In contrast, the student model is trained on grayscale data to address resource constraints and real-time requirements. We investigate two training strategies: direct cross-modal knowledge distillation and a two-step process where the student model is first independently trained and then refined with knowledge distillation. Our findings indicate notable improvements in efficiency and accuracy, demonstrating the potential of our approach to constructing lightweight trained image classification models for resource-limited environments. Moreover, the proposed solution can be adapted to any computer vision image classification scenario and in this work, the EDEN dataset was used for proof of concept.