Segmentation-Assisted Diabetic Retinopathy Classification Using Hybrid ResNet-ViT and Correlated Feature Integration

: Accurately detecting retinal lesions for evaluating the progression of diabetic retinopathy (DR) is still a challenging and laborious task in medical imaging. The disease often progresses without noticeable warning signs, making early detection challenging. Most of the computer-aided diagnostic systems for DR grading utilize deep learning models without segmentation, which is essential for obtaining accurate results. We developed a segmentation-assisted DR classification using a hybrid ResNet-ViT model. The pipeline is well structured comprising lesion segmentation using modified MesU-Net, a hybrid feature extraction using ResNet50 and vision transformer (ViT), feature fusion using canonical correlation analysis (CCA), and traditional machine learning (ML) classifiers for DR grading. The segmentation of the lesion was initially performed in this DR classification method using the modified MesU-Net model, which focuses on highlighting the retinal characteristics based on the lesion that are essential for precisely identifying the stage of the disease. This integrated model was designed to extract significant features from both normal and segmented retinal image data. For feature fusion, CCA was used to pinpoint and extract the most highly correlated features from these distinct data views. These robustly correlated features were then passed to traditional machine learning classifiers for DR grading. The proposed model was evaluated using the fine-grained annotated DR dataset for lesion extraction and classification. The experimental results demonstrate that the ResNet-ViT network combined with a support vector machine classifier delivers the best performance. The proposed method achieved an average accuracy of 97.6 % for DR grading, highlighting its effectiveness in classifying DR severity.