S482 A Multimodal Vision Transformer Model for Real-Time Histopathological and Morphological Classification of Colonic Polyps

Introduction: Colorectal cancer is the second leading cause of cancer-related mortality worldwide, claiming over 900,000 lives annually. Automated deep learning for polyp detection has improved accuracy, but most systems rely solely on visual features and omit key clinical metadata. Methods: We extracted video frames at 2 frames per second from the ERC PMP-v5 colonoscopy dataset and aligned each frame with corresponding high-resolution still images. We linked every image or frame to structured clinical metadata—patient age, polyp size, dysplasia grade, and differentiation status—via a unique patient identification. Continuous variables underwent scaling and categorical fields received 1-hot encoding. Parallelly, we resized images to 224 × 224 pixels, normalized pixel intensities, and passed them through a pre-trained Vision Transformer (ViT) to generate 768-dimensional feature vectors. We concatenated these visual embeddings with the processed metadata to form a unified input, then trained 2 Random Forest classifiers—one for histopathological prediction and another for morphological classification. All 217 patients were partitioned at the patient level into training (60%), validation (20%), and testing (20%) cohorts to ensure strict separation and to prevent data leakage. Results: On the independent test cohort, the pathology classifier achieved 99.05% overall accuracy in distinguishing histological types (Tubular, Villous, Serrated, Adenocarcinoma), and the morphology classifier reached 94.94% accuracy for Paris classification subtypes (e.g., 0-IIa, 0-Ip). Both models demonstrated stable performance across patient subgroups, with no evidence of overfitting on the validation data. The fusion of image features and clinical metadata yielded robust, real-time predictive capability suitable for integration into endoscopic workflows. Conclusion: By fusing Vision Transformer–derived image embeddings with encoded patient metadata, our model achieved 99% accuracy for histopathological prediction and 95% accuracy for Paris-based morphology classification on an independent test set. This multimodal approach can standardize polyp assessment, reduce inter-observer variability, and deliver real-time decision support during colonoscopy. Prospective validation and seamless integration into endoscopy platforms are the next critical steps.