Fine-Tuning Depth Analysis: Identifying the Sweet Spot for Maximum Accuracy in CNNs

Convolutional Neural Networks (CNN) have transformed the field of computer vision through their exceptional performance in image classification, face recognition, and object detection. Much of its success is due to the development of transfer learning, where pre-trained models trained on large datasets such as ImageNet are fine-tuned (adapted) for new tasks, even when only limited labeled data is available. Fine-tuning involves gradually unfreezing and retraining a pre-trained model to obtain optimal accuracy. However, determining the optimal number of layers to unfreeze remains a critical challenge. Unfreezing too few layers may limit the model's ability to adapt to task-specific features, leading to under-fitting, while fine-tuning too many layers risks over-fitting, thereby compromising generalization on unseen data. This research aims to systematically determine the number of layers to fine-tune in a pre-trained CNN to achieve optimal performance. Grad-CAM and experimental approach, which involves fine-tuning ResNet152, EfficientNetB0, and VGG16 networks, are used. Findings show that unfreezing one-fifth (20% to 25%) of the top layers gives optimal performance and unfreezing too many layers leads to overfitting. However, the VGG16 network requires unfreezing the entire layers for optimal performance because of its few layers (18 layers). Future research can consider other pre-trained networks to ascertain the findings of this study. This research is significant to AI researchers, AI engineers, data analysts, and individuals who build AI systems.