S1301 AI-Powered Lesion Detection and Localization for AR-Enabled Capsule Endoscopy: A Step Toward Real-Time Clinical Decision Support

Introduction: Timely identification of gastrointestinal lesions improves treatment planning and patient outcomes. Variability in image quality, lesion morphology, and illumination challenges traditional methods. We propose a dual‐model framework that combines a Vision Transformer (ViT) for classification with a U-Net for precise lesion segmentation. This system targets real-time deployment in augmented reality (AR)-guided endoscopy. Methods: We curated 18,432 labeled images and 23,156 lesion instances from the KYUCapsule dataset. For classification, we fine-tuned a ViT-Base model pretrained on ImageNet. We resized input frames to 224 × 224 pixels and applied brightness shifts, contrast normalization, and small rotations for data augmentation. We replaced the original classification head with a dense layer optimized for 3 lesion categories (inflammatory, vascular, neoplastic). Training employed AdamW optimization (initial learning rate 1 × 10⁻⁴, weight decay 1 × 10⁻⁵) with a cosine annealing schedule. We implemented early stopping to prevent overfitting. For segmentation, we built a U-Net with a ResNet-34 encoder, training it on expert-annotated binary masks. Random horizontal and vertical flips enhanced generalizability. Both models trained on a stratified 70%:15%:15% split, ensuring equal representation of lesion types. We tuned batch size (32–64), dropout (0.2–0.5), and learning rate decay schedules on the validation set. Experiments ran on NVIDIA Tesla V100 GPUs with mixed-precision training. Results: On the held-out test set (2,764 classification images; 3,474 segmentation masks), ViT achieved an overall accuracy of 92.4%, precision of 91.8%, recall of 92.1%, and F1-score of 92.0%. Subgroup performance remained consistent: inflammatory 93.1%, vascular 91.5%, neoplastic 92.2%. Attention heatmaps demonstrated focus on relevant morphological features. The U-Net segmentation model reached a Dice coefficient of 91.3% and an IoU of 90.5%, accurately delineating lesion boundaries even in low-contrast regions. Combined inference time averaged 45 milliseconds per frame, meeting AR real-time requirements. Calibration analysis yielded a Brier score of 0.083, indicating minimal overconfidence. Conclusion: Our transformer-based approach accurately classifies and segments gastrointestinal lesions in real time. High performance, rapid inference, and interpretability support integration into AR-guided endoscopy workflows. This system promises to enhance diagnostic precision and improve patient care.