Classification of metal surface defects using a hybrid deep learning and machine learning approach

This study presents a comprehensive hybrid deep learning and machine learning framework for steel surface defect classification and evaluates its robustness under controlled, augmented, and external benchmarking conditions. Six convolutional neural network backbones MobileNet-V2, ResNet-50, DenseNet-121, EfficientNet-B0, U-Net, and LSDNet were evaluated using both end-to-end deep learning and hybrid classification with Logistic Regression, Support Vector Machine, Decision Tree, and K-Nearest Neighbors. Experiments were conducted on the original NEU dataset containing 1,800 images from six classes, a source-disjoint augmented NEU dataset containing 3,600 images, and the GC10-DET dataset containing 2,312 images from ten classes as an independent external benchmark. Performance was assessed using five-fold cross-validation, out-of-fold predictions, accuracy, balanced accuracy, macro precision, macro recall, macro F1-score, Matthews Correlation Coefficient, Cohen’s Kappa, ROC-AUC, bootstrap confidence intervals, and statistical significance tests. On the original NEU dataset, eight of the 30 backbone–head combinations achieved perfect OOF accuracy, while all configurations exceeded 0.9867, indicating benchmark saturation. Augmentation mainly benefited weaker configurations, with the largest improvement observed for LSDNet + Decision Tree (+ 0.0089). In contrast, GC10-DET produced substantially greater performance variation. DenseNet-121 + Logistic Regression achieved the highest overall accuracy (0.8984), whereas DenseNet-121 with the end-to-end deep learning head achieved the highest macro F1-score (0.8681) and balanced accuracy (0.8609). Statistical analyses further showed that differences among classifier heads were more pronounced on GC10-DET than on NEU. Overall, hybrid classifiers can match end-to-end deep learning on highly separable benchmarks, whereas end-to-end representations become more important under class imbalance, visual overlap, and increased distributional complexity. Grad-CAM, Grad-CAM++, LIME, and SHAP provided complementary qualitative explanations of the image regions contributing to model predictions.