MORSE: Leveraging Morphological and Spectral Priors for Early Disease Detection

In medical image analysis, transfer learning from ImageNet-pretrained models is a dominant paradigm, but its effectiveness on small clinical datasets is questionable. In this work, we challenge this understanding by showing that paradigms for natural image encoders can fundamentally fail to suit the medical imaging context for early disease detection, where data are scarce. We present MORSE (Morphological Observation with Reweighted Spectral Encoding), a minimal architecture that incorporates specific inductive biases by performing multi-scale morphological operations, adaptive gating of the spectrally decomposed signals, and memory-efficient hierarchical fusion. MORSE is evaluated on four clinical medical imaging tasks from dermatology, ophthalmology and neurology, and shows consistent improvements over eight state-of-the-art (SoTA) baselines trained from random initialization, as well as ImageNet-pretrained models and a RadImageNet-pretrained ResNet-50 (the only RadImageNet backbone tested). With limited training data, MORSE outperforms the strongest from-scratch baselines by 0.83 to 5.00 percentage points and exceeds the best pretrained models by 0.23 to 2.27 percentage points, whereas pure vision transformers underperform by approximately 16 to 45 percentage points on the three smaller datasets. Ablation studies support the complementary contributions of morphological and spectral features, and parameter analysis shows that architecture design using medical imaging priors can be more consequential than model capacity in the evaluated small-data settings. These studies challenge the prevailing reliance on transfer learning in data-constrained medical imaging, suggesting that domain-specific architectural innovation is a viable alternative for specialized diagnostic tasks.