S1299 Video-Based Gastrointestinal Lesion Detection Using Deep Temporal Modeling with 3D CNN

Introduction: Accurate classification of gastrointestinal lesions during endoscopic examinations is essential for early diagnosis and appropriate treatment. Traditional computer-aided detection systems focus on single-frame analysis and overlook the continuity inherent in video streams. We implemented a deep-learning framework that combines spatial embeddings from a pretrained Vision Transformer with 3-dimensional convolutional processing to capture both morphological and temporal cues in endoscopic video sequences. Methods: We obtained over 1 million annotated video frames from the HyperKvasir repository. We segmented each endoscopic examination into overlapping sequences of 16 frames (stride 4), yielding more than 7,500 temporal samples. We extracted 768-dimensional features per frame using a pre-trained Vision Transformer and reshaped these into volumetric tensors of size 16 × 32 × 24 to preserve spatial and temporal context. We implemented a 3-dimensional convolutional neural network enhanced with batch normalization, ReLU activations, and dropout, and optimized it using the Adam algorithm with a stepwise learning rate scheduler across 500 epochs. Data were partitioned into training (80%), validation (10%), and independent test (10%) sets, with stratified sampling to mitigate class imbalance. Results: On the held-out test set, the network achieved 97.2% classification accuracy and a weighted F1-score of 0.97. The weighted-average area under the receiver operating characteristic curve reached 1.00, reflecting excellent discrimination among lesion types. Confusion matrix analysis confirmed high precision and recall across all major categories despite imbalanced class distributions. Dimensionality reduction via t-distributed stochastic neighbor embedding revealed well-defined clusters corresponding to distinct gastrointestinal lesion classes. Conclusion: The network delivered 97.2% accuracy, a weighted F1-score of 0.97, and an AUC-ROC of 1.00 on an independent test dataset. Robust results across lesion categories and distinct clustering in t-SNE embeddings underscore its reliability. By integrating spatial-temporal features, this approach holds promise for real-time decision support and may reduce diagnostic variability in gastrointestinal endoscopy.