Cross-modal attention fusion of ultrasound imaging and clinical tabular data for breast lesion classification: A multimodal deep learning framework

Abstract Accurate classification of breast lesions in ultrasound images is an important yet challenging task, plagued by visual ambiguity between classes and morphological variability within classes. This paper presents a multimodal deep learning framework combining EfficientNet-B4 visual features with an 11-dimensional, leakage-free clinical tabular feature vector, organized into three feature-group tokens (intensity, texture, and geometry), through a multi-token cross-modal attention fusion mechanism. A systematic audit of candidate tabular features identified that eight commonly used lesion-shape descriptors (area, width, height, aspect ratio, solidity, eccentricity, and mask-region intensity statistics) are only defined when a segmentation mask exists, and their availability, not merely their value, correlates with class label; these features are excluded from the tabular branch, retaining only whole-image intensity and texture statistics defined identically for every image. On the BUSI breast ultrasound dataset (780 samples: 437 benign, 210 malignant, 133 normal) evaluated by five-fold stratified cross-validation, the tabular-only ablation using this leakage-free feature set achieves 66.72% macro AUC and 51.28% accuracy, well below both the image-only ablation (86.92% AUC) and every multimodal fusion configuration tested, indicating limited independent diagnostic signal in the retained tabular features alone. The proposed multi-token cross-modal attention mechanism achieves 95.46% macro AUC and 88.59% accuracy, comparable to, but not exceeding, a plain concatenation baseline (96.28% AUC) and two of three additional fusion strategies evaluated (additive fusion, 96.00%; bilinear fusion, 95.82%). All differences among the fusion strategies tested fall within approximately one fold-to-fold standard deviation of one another, indicating no statistically meaningful separation on this dataset at this sample size. We report the tabular-feature leakage audit as the primary methodological contribution of this study, applicable to multimodal medical imaging pipelines more broadly, and report the fusion-strategy comparison directly rather than selectively emphasizing the proposed attention mechanism.