Multimodal pill image recognition in natural environments using context-aware cross-modal attention

Medication errors remain a major source of preventable harm, and identifying a pill from a photograph taken in an uncontrolled setting is still difficult because the informative cues-color, shape, surface texture, and imprinted text-are unequally reliable from one image to the next. We present a recognition framework for natural environments in which a fusion module, context-aware cross-modal attention (CACMA), weighs these four modalities separately for every image using a query formed from their joint global context, an explicit confidence branch that estimates per-modality reliability, and a three-way integration of attended, confidence-weighted, and residual features, on top of \(\hbox {U}^2\) -Net segmentation (98.06% validation IoU) and connected-component localization. All results follow a unified end-to-end protocol, with segmentation, localization, OCR, and fusion executed at inference time, over three random seeds on two real-world datasets. On CURE (196 classes) CACMA attains 91.13±0.27% Top-1 accuracy and \(95.50\pm 0.16\%\) mAP, exceeding the published results of Ling et al. and Rádli et al. by 22.83 and 5.41 pp and summation fusion by 3.15 pp ( \(p<0.001\) ); the 0.76 pp margin over weighted summation is not statistically significant, which we report as parity rather than superiority, while CACMA attains the smallest across-seed variability of any method evaluated ( \(\pm 0.27\) pp). On OGYEI-v2 (112 classes) it reaches \(83.23\pm 0.73\%\) Top-1 accuracy. What fixed-weight fusion cannot reproduce is directly measurable: the confidence assigned to the text modality rises from 0.362 to 0.602 when an imprint is legible ( \(p = 1 \times 10^{-55}\) ), with the released weight redistributed to the visual modalities. A fully fine-tuned ViT-B/16 is not significantly more accurate ( \(92.07\pm 0.64\%\) ; McNemar \(p=0.32\) ) while using 6.4 \(\times \) the parameters and offering no modality-level explanation, placing CACMA at a favorable accuracy-efficiency-interpretability operating point. Paired significance testing applied to the ablations shows that no individual component accounts for the result on its own under this segmentation quality; together with calibration, saliency and failure analyses, these tests delimit where multimodal attention fusion helps and where it does not.