A hybrid CoordAttention model with focal loss for handling visual similarity in gastrointestinal endoscopic images

Gastrointestinal (GI) diseases represent a major global health challenge that requires timely and accurate diagnosis for effective treatment. Deep learning has shown promise in endoscopic image analysis. However, challenges such as class imbalance and visual similarity among classes degrade the prediction performance. This study proposes a hybrid deep learning framework that combines ResNet50 and VGG16 architectures with the Coordinate Attention (CoordAttention) mechanism to enhance feature representation and improve classification accuracy. Both architectures were individually optimized with CoordAttention before concatenating them into a feature-level hybrid model, which was further refined using focal loss to address class imbalance. Experiments on the 22-class GastroVision dataset showed that the proposed hybrid with CoordAttention achieved higher scores than the baseline and the compared state-of-the-art models, with a macro F1-score of 0.718, a micro F1-score of 0.897, and a Matthews correlation coefficient (MCC) of 0.886. The hybrid model without CoordAttention obtained a macro F1-score of 0.657, a micro F1-score of 0.835, and an MCC of 0.816 with the focal loss function. Focal Loss mitigates the effects of class imbalance by emphasizing difficult and underrepresented samples. The proposed approach did not apply any synthetic technique to handle imbalance, which may make it applicable to datasets with different class distributions. Although standard data augmentation was applied uniformly to improve the diversity of datasets, no specific class-balancing strategy was used to artificially balance class distributions. These findings suggest that the proposed framework is a useful approach for jointly addressing class imbalance and visual similarity in GI endoscopy image classification.