Evaluating Knowledge Distillation and Compression Techniques for Edge Class Devices

Recent advances in Machine Learning (ML) technologies have made it possible to execute neural network models in daily devices such as smartphones and IoT.Due to these advancements, intelligent functionalities can now be integrated into devices with limited resources, enabling AI capabilities like speech recognition, and image processing.However as the AI models go deeper, hardware requirements become a substantial issue since the number of computations increases drastically.To mitigate this, researchers discovered ways to compress AI models without sacrificing a lot of performance.Some examples are quantization and knowledge distillation.These techniques aim to reduce the size, and computational complexity, of the AI models, allowing them to be deployed more efficiently on hardware with limited resources.In this research paper, we evaluate these methods and compare them in terms of efficiency and accuracy.The results showed that knowledge distillation can can be applied in real-life scenarios.