S3091 Multimodal Transformer Model Predicts Post-Treatment Lesion Recurrence in Gastrointestinal Patients

Introduction: Accurate prediction of gastrointestinal lesion recurrence after treatment remains a critical unmet need. Integrating visual features from endoscopic imaging with patient clinical data may enhance risk stratification. We developed and evaluated a multimodal deep‐learning pipeline combining Vision Transformer embeddings of capsule endoscopy images with structured metadata to predict lesion recurrence. Methods: We retrospectively assembled a cohort of 1,200 adult patients from the KYUCapsule dataset who underwent capsule endoscopy and follow-up for at least 12 months. We annotated 3,600 high-resolution lesion frames, categorizing each by lesion type (inflammatory, vascular, or neoplastic) and extracting corresponding metadata: age, sex, treatment regimen, lesion classification, history of prior recurrence, and follow-up duration. Image frames passed through a pre trained Vision Transformer (ViT), with final layers fine-tuned on our dataset to produce 512-dimension feature vectors capturing morphology and texture. Simultaneously, we encoded structured metadata via a fully connected neural network (128 and 64 neurons, ReLU activation). We concatenated image and metadata embeddings and trained a multilayer perceptron classifier with 2 hidden layers (256 and 128 neurons) and dropout (rate = 0.3). We split data into training (70%), validation (15%), and test (15%) sets, ensuring equal representation of recurrence vs non-recurrence cases through oversampling. Hyperparameters (learning rate, batch size, L2 regularization λ = 0.001) were optimized via grid search on the validation set. We used cross-entropy loss, Adam optimization, and early stopping. Results: On the test set (n = 180), the multimodal model achieved accuracy = 89.2%, precision = 88.7%, recall = 88.5%, and F1-score = 88.9%. The area under the ROC curve reached 0.902. Subgroup analysis showed consistent accuracy across lesion types: inflammatory = 90.5%, vascular = 88.0%, and neoplastic = 89.8%. Calibration was strong (Brier score = 0.087). SHAP analysis identified lesion type, prior recurrence history, and image-derived severity metrics as the top 3 predictors. Conclusion: Our multimodal transformer-based approach reliably integrates endoscopic imaging and clinical metadata to predict gastrointestinal lesion recurrence. Given its high discriminative performance and interpretability, this model could aid clinicians in tailoring surveillance and management strategies, potentially improving patient outcomes.