Self-supervised learning outperforms supervised learning for crop classification by annotating only 5% of images

Abstract Purpose One of the most pervasive Artificial Intelligence (AI) methodologies utilized in the domain of agriculture for image-based classification purposes is Supervised Learning (SL). However, SL depends on a large amount of annotation effort and is susceptible to overfitting to the given prediction task. Self-Supervised Learning (SSL) is a novel training paradigm with the potential to address these issues, while its potential has not been investigated in the agriculture domain. This paper presents the initial experimental investigation and comparison of SL and SSL for the classification of agricultural images in the context of limited samples. Methods We used an agricultural subset of the Land Use and Cover Area Frame Survey (LUCAS) dataset serving as a case study. In total, it comprised 1,000 images for each of the 10 crops: common wheat, barley, oats, maize, potatoes, sugar beet, sunflower, rape and turnip rape, soya, and temporary grassland. For SL, we trained popular and frequently used Convolutional Neural Network (CNN) architectures such as VGG16, Inception, ResNet-18/50, SqueezeNet, ResNeXt-50, MobileNet-V2, ShuffleNet, EfficientNet-V2, and ConvNeXt Tiny with and without data augmentations. For SSL, the best-performing CNN architectures (ResNet-18, ResNet-50, and ResNeXt-50) were further tested. The architectures were pre-trained with the VICReg algorithm (Variance Invariance Covariance Regularization) and fine-tuned successively using supervision for crop type classification. Results Our results demonstrate that the SSL models can distinguish crop types (common wheat, barley, oats, maize, potatoes, sugar beet, sunflower, rape, soya, and grassland) even without labels based solely on morphological features and organize them into three semantically meaningful visual groups: cereal-like and grassland crops, upright broadleaf crops, and low-growing broadleaf crops. The fine-tuned models, particularly ResNeXt-50, achieved superior performance compared to any of the SLs. Notably, we show that the fine-tuned SSL models outperformed the best-performing SL models by using only 5% of the labeled training data for fine-tuning, corresponding to a small and balanced subset of the training split. Conclusion These findings highlight the potential of SSL for improving classification efficiency and generalization under limited data availability conditions in agriculture applications, providing a viable path toward more efficient agricultural monitoring systems.