Beyond Accuracy: Calibration and Uncertainty in Multi- Architecture Ensembles for MedMNIST Classification

An accurate prediction is not the same as a trustworthy one. In medical image classification, most benchmarks rank models on discrimination alone, yet a confidently wrong output and a flagged one carry very different clinical risk. This work addresses that gap by evaluating four architecturally diverse deep networks, ConvNeXt-Base, Vision Transformer Base (ViT-Base), EfficientNetV2-M and InceptionResNetV2, together with two ensemble strategies, Soft Voting and Rigorous Stacking, across four MedMNIST v2 modalities used at 224 × 224 resolution: eight-class blood cell microscopy (BloodMNIST), binary breast ultrasound malignancy detection (BreastMNIST), seven-class dermatoscopic lesion classification (DermaMNIST), and eleven-class abdominal computed tomography (CT) organ identification (OrganAMNIST). All models were fine-tuned from ImageNet weights. The contribution is a joint, multi-modality evaluation along three layers that the MedMNIST literature has not previously combined. Discrimination is reported as accuracy, area under the receiver operating characteristic curve (AUC), and macro-averaged precision, recall, the F1 score and the area under the precision-recall curve (PR-AUC), the macro average chosen to respect the documented class imbalance. Predictive uncertainty is summarised through the Shannon entropy of correct against incorrect predictions. Calibration is summarised through reliability diagrams and the expected calibration error (ECE), supported by gradient-weighted class activation mapping (Grad-CAM) and attention-based interpretability for the two model families. Every trained configuration exceeded the reported benchmark accuracy on all four datasets, with margins ranging from a fraction of a percentage point on the near-saturated BloodMNIST task to roughly fifteen percentage points on DermaMNIST. The ensembles matched or exceeded the strongest individual model in almost every case, the single exception being Rigorous Stacking on the small BreastMNIST set. The central finding is that accuracy and calibration are dissociated: the most accurate configuration is rarely the best calibrated. Soft Voting attained the lower ECE on all four datasets and the higher accuracy on three, whereas the accuracy advantage of Rigorous Stacking was confined to the most imbalanced dataset, DermaMNIST, and came there at the cost of weaker probabilities. Predictive entropy placed incorrect predictions at higher uncertainty than correct ones throughout, which is the precondition for confidence-based triage, but the usable separation between the two distributions was held most consistently by Soft Voting, since a strong aggregate calibration score could still mask a collapsed, non-informative per-prediction uncertainty. Overconfidence was pervasive among the individual models and was reduced, but not removed, by ensembling. Residual errors concentrated at clinically recognised decision boundaries, including the melanoma to nevi distinction in dermoscopy, malignant false negatives in ultrasound, and kidney laterality in CT, and even the largest accuracy gain left roughly a quarter of melanomas assigned to benign nevi. The analysis therefore argues that ensemble methods for medical imaging should be judged by the calibration and the structure of their predictive uncertainty as much as by discrimination accuracy, that post-hoc calibration is a prerequisite for safe deployment rather than an optional refinement, and that Soft Voting is the more dependable clinical default. These results derive from single-source benchmark data at 224 × 224 and a single training run under one fixed seed, so higher-resolution and prospective external validation remain the necessary next steps toward deployment.