Dual-stream EfficientNet–Swin Transformer architecture with multi-head attention fusion for endoscopic gastrointestinal image classification

Automated gastrointestinal lesion recognition from endoscopic imagery is a clinically demanding task that simultaneously requires sensitivity to fine-grained mucosal texture and awareness of broader spatial organisation within the field of view. Conventional single-backbone networks address one of these requirements well but rarely both, limiting their practical utility in computer-aided colonoscopy workflows. To bridge this gap, we introduce a dual-stream architecture that couples EfficientNet-B3—selected for its parameter-efficient compound-scaling strategy—with the Swin Transformer, whose hierarchical shifted-window self-attention encodes multi-scale contextual dependencies without the quadratic complexity of standard vision transformers. Rather than merging the two streams through static concatenation or simple average pooling, we design a Multi-Head Attention Fusion (MHAF) module that learns, end-to-end, how much each stream should contribute at each spatial position and for each sample. This adaptive gating eliminates redundancy introduced by naive fusion while preserving cross-modal complementarity. The complete pipeline is trained on the eight-class Kvasir endoscopic benchmark under a rigorous experimental protocol. This protocol integrates stratified splitting, aggressive data augmentation, and model selection governed by the Matthews Correlation Coefficient (MCC) a metric particularly well-suited to imbalanced multi-class settings. On the held-out test partition, the framework achieves 95.4% accuracy and an MCC of 0.949. Structured ablation experiments quantify the incremental value of each design decision: the MHAF module alone accounts for a 2–3 percentage-point improvement over the best static-fusion baseline. Cross-dataset evaluation on CVC-ClinicDB and LC25000 suggests that the learned representations retain discriminative capability across related gastrointestinal imaging domains without additional fine-tuning. Grad-weighted Class Activation Maps reveal that the model’s attention consistently falls on pathologically meaningful structures—lesion margins, vascular patterns, and mucosal irregularities—rather than on imaging artefacts, supporting clinical trustworthiness. With an inference latency of 55 ms per frame on a single GPU, the proposed framework is compatible with real-time colonoscopy assistance, offering a concrete step toward reducing inter-observer variability and diagnostic workload in colorectal cancer screening. Cross-dataset evaluation on CVC-ClinicDB and LC25000 confirms that the learned representations transfer across imaging modalities and tissue types without additional fine-tuning.