A comparative study of hybrid approaches for diabetic retinopathy severity grading

Abstract Diabetic retinopathy (DR) is a progressive ocular complication of diabetes and is commonly graded into five severity levels, ranging from no DR to proliferative DR. Automated severity grading is challenging because the classes are imbalanced and misclassification between severity levels may have different implications depending on the distance between the true and predicted grades. This study presents a comparative evaluation of hybrid approaches for DR severity grading using the APTOS 2019 dataset. The evaluated framework configurations investigate the effects of Wavelet Morphological pre-processing, DenseNet121-based feature extraction, LSTM-based feature transformation, XGBoost classification, severity-based weighting, and ordinal learning. A stratified five-fold cross-validation protocol was used, with all comparative models evaluated using consistent out-of-fold predictions. In addition to conventional classification measures, the study reports quadratic weighted kappa (QWK), mean absolute error (MAE), root mean squared error (RMSE), adjacent accuracy, class-wise and macro-area under the receiver operating characteristic curve (AUC), and error-distance distributions. The complete hybrid configuration achieved an accuracy of 78.32%, macro-F1 score of 0.577, QWK of 0.825, and MAE of 0.319. Comparative experiments demonstrated that simpler configurations based on DenseNet121 and conventional XGBoost classification achieved higher performance on several evaluation measures, while the standard ordinal CORAL model provided competitive ordinal performance with improved calibration. These findings indicate that the contribution of individual preprocessing, feature-transformation, and classification components should be evaluated independently rather than assuming that increased architectural complexity necessarily improves DR severity grading. The study provides a systematic comparison of hybrid and ordinal approaches and highlights the challenges of accurately distinguishing the less represented advanced DR severity grades.