Beyond Accuracy Rankings: Cross-Dataset Stability and Computational Cost in Plant Disease Classification

Abstract On a specific dataset, achieving high accuracy in plant disease classification with CNN is possible, but it should also be evaluated on other datasets and the accuracy of the CNNs depend on the cost as well. This study tries to compare the performance of ResNet-50, MobileNetV2 and EfficientNet B0 architectures with frozen feature transfer learning approach. The 38-class New Plant Diseases Dataset (Augmented) has been used to train and internally validate models consisting of 70,295 images from which 56,236 have been used to train the models and 14,059 used for tests. A total of 146 images of the PlantDoc were used for external evaluation for 17 classes. The accuracy, macro-F1, and weighted-F1 score were used to measure the classification performance, while the number of the parameters, batch-one inference latency and throughput were used as measurement for the computational efficiency. ResNet-50 got the best internal accuracy score (96.33%) and macro-F1 (96.22%). The best accuracy (21.23%) was obtained by ResNet-50 while the best macro-F1 (22.48%) and measured inference latency (4.97 ms) was obtained by MobileNetV2 on PlantDoc. In terms of measured throughput, MobileNetV2 had the lowest number of parameters (2.27 million) and the highest throughput of 201.2 images/s. The pairwise McNemar tests did not reveal any statistically significant differences with respect to the prediction of correctness. There is a restriction of the external sample and hardware specific benchmarking leading to limitation of generalization. The results suggest that it is worthwhile to combine a measure of predictive performance with a measure of computational efficiency.