S1300 Automated Classification of Gastrointestinal Endoscopic Findings Using a Fine-Tuned Vision Transformer

Introduction: Clinicians performing gastrointestinal endoscopy face considerable challenges when evaluating a wide spectrum of mucosal findings and subtle lesions in real time. Inter-observer variability and the volume of procedures can delay accurate diagnosis and affect patient outcomes. Automated image classification using advanced deep-learning architectures offers a promising route to standardize lesion detection and streamline endoscopic workflow. Methods: We processed 374 anonymized clinical endoscopy recordings and extracted 15 representative frames per video by applying K-means clustering to feature embeddings generated by a pretrained Vision Transformer. Expert endoscopists assigned each of the resulting 5,296 frames to 1 of 29 gastrointestinal diagnostic categories, covering pathological findings (for example, polyps and ulcers), anatomical landmarks, and mucosal quality assessments. We fine-tuned the ViT backbone for 10 epochs using class-balanced cross-entropy loss and the Adam optimizer (learning rate 3 × 10⁻⁵, batch size 32). Data were split into 80% for training, 10% for validation, and 10% for testing, with early stopping triggered by lack of improvement in validation loss to prevent overfitting. Results: Fine-tuning achieved 90.0% overall accuracy on the held-out test set. Top-3 and Top-5 accuracies reached 98.2% and 99.0%, respectively. The weighted F1-score of 0.89 reflected balanced performance across all diagnostic categories. Receiver operating characteristic analysis yielded area under the curve values above 0.98 for 25 of the 29 classes, demonstrating strong discrimination. The confusion matrix indicated minimal misclassification, largely confined to visually similar lesions and adjacent anatomical landmarks. Conclusion: Fine-tuned Vision Transformer models achieved high overall accuracy (90.0%), exceptional top-5 performance (99.0%), and strong discrimination across 29 diagnostic categories. Embedding this approach into endoscopic systems can support rapid, reliable lesion identification, reduce diagnostic variability, and enhance clinical efficiency. Further validation in prospective clinical trials will clarify its impact on patient care.