Knowledge Distillation for Resource Efficient Classification with CLIP-Guided Specialist Networks

The complexity of training datasets which are characterized by a large number of classes with subtle inter-class differences, poses significant challenges for model efficiency and deployment on resource constrained environments. While large deep learning models excel at capturing these complex class structures, their computational demands make them impractical for deployment on resource-constrained devices. In contrast, smaller models, though more efficient, struggle to achieve comparable performance due to their limited capacity. Knowledge distillation helps bridge this gap by transferring knowledge from a large, powerful teacher model to a smaller, more efficient student model. In this paper, we introduce a specialized knowledge distillation method. Instead of using a single student model, we train multiple specialist models, each focusing on a subset of classes. This helps the models better distinguish between similar classes. Our approach automatically groups classes into meaningful clusters by averaging CLIP embeddings at the class level. We also utilize a cluster predictor that dynamically selects the most suitable specialist(s) for each input, balancing accuracy with efficiency. We test our method on ImageNet and show that these specialist models outperform traditional student models on their respective class subsets. Our approach provides a scalable way to build accurate and efficient models suitable for resource-constrained environments.