S1303 Multimodal Classification of Gastrointestinal Conditions Using Vision Transformer-Based Deep Feature Extraction

Introduction: Accurate classification of gastrointestinal (GI) conditions via endoscopic imaging remains a diagnostic challenge, particularly in cases with subtle or overlapping features. Advances in artificial intelligence, particularly transfer learning with vision transformers (ViTs), offer promising solutions for automated, high-fidelity image interpretation across diverse GI pathologies. Methods: We developed a multimodal pipeline to classify gastrointestinal (GI) conditions using the HyperKvasir dataset, which includes 10,662 labeled images (23 classes), 1,000 segmented polyp images with pixel-level masks, and 373 video recordings (over 1 million frames). All images were preprocessed and normalized. Visual features were extracted using a Vision Transformer (ViT) pretrained on ImageNet. For segmented images, features were derived from both the originals and their masks. For videos, frame-wise features were aggregated. These vectors were input into a custom multi-layer perceptron (MLP) classifier for multiclass prediction. The model was trained using the Adam optimizer with cross-entropy loss and dropout regularization. Performance was evaluated on a test set of 2,133 samples using accuracy, macro F1-score, Top-K accuracy, and class-wise area under the ROC curve (AUC). Results: The model achieved 88.61% accuracy and an 88.34% macro F1-score, showing strong performance across both common and rare classes. Top-3 and Top-5 accuracies reached 97.19% and 98.59%, respectively. Most classes had ROC-AUC >0.94, with AUC = 1.00 for conditions such as esophagitis, Barrett’s esophagus, and ulcerative colitis. A confusion matrix and multiclass ROC plots demonstrated consistent diagnostic performance and highlighted the model’s ability to distinguish subtle visual features across GI pathologies. The system’s reliability across varied modalities supports its potential integration into clinical endoscopic workflows. Conclusion: This ViT-based classification system demonstrates high accuracy and robustness across static, segmented, and video-based GI imaging. Its performance and generalizability suggest strong potential for clinical application as a decision-support tool in endoscopic diagnostics.