A Hybrid Gabor–ViT Embedding with Gated Mamba Deep Learning Framework for Brain Tumor MRI Classification

Brain tumor classification from MRI is an important task in medical image analysis. This study introduces a new hybrid deep learning method that integrates Gabor filtering for the improvement in texture feature extraction, the Vision Transformer (ViT) framework for extraction of the global context dependency and fine-grained spatial features, and a novel Gated Mamba to enhance feature classification efficiency. The framework was first evaluated in-distribution on the Kaggle dataset, which contains 5712 slices of brain MRI images from four different classes using accuracy, precision, recall, F1-score, AUC, Std, and 95% confidence intervals (CIs), with a mean image-level accuracy of 99.62%. Because the Kaggle dataset is known to be near-saturated and no patient identifiers are available, this in-distribution result is treated as a sanity check rather than a measure of clinical generalization; pHash deduplication was used to remove visually identical images, but this does not provide patient-level grouping, and residual patient-level leakage cannot be fully excluded. The central evaluation of this study is therefore a cross-dataset validation on the “figshare brain mri” dataset, which uses a different scanner and acquisition protocol and was never used during training and model selection; the proposed model on this dataset achieves 87.23% accuracy with reasonable confidence intervals, a decline of 12.39 percentage points relative to the in-distribution result that reflects the domain-shift gap typical of this task. The proposed hybrid approach performs competitively against established methods in-distribution, while the cross-dataset results highlight generalization challenges that motivate the failure analysis and limitations discussed in this paper.