Beyond Accuracy: A Clinically-Oriented Breast Cancer Diagnostic Framework Integrating Conformal Prediction, Counterfactual Explanations, and Multi-Task Learning on FNA Cytological Features

Breast cancer remains a serious global health threat, and there is a clear need for diagnostically reliable research frameworks that go beyond the overly optimistic performance claims often reported in the literature. This study provides a clinical diagnostic methodology for breast cancer using fine-needle aspiration cytology, which is intended to promote clinical validation beyond benchmark maximization. The methodology presented uses Wisconsin Diagnostic Breast Cancer and Wisconsin Prognostic Breast Cancer datasets to develop a completely leak-proof nested 5x5 cross-validation procedure where any preprocessing was limited to training folds only. The original 30 cytology features were increased to 42 by adding biologically-motivated feature engineering, and a stacked ensemble method was trained to provide clinically interpretable probability estimates. The method used bootstrap-based feature stability, split conformal prediction to perform uncertainty-driven triage, feature attribution and counterfactual explanation modules, decision curve analysis for clinical validation, and multi-task learning for cross-dataset transfer analysis. In addition, the method yielded an out-of-fold area under the curve of 0.9932 and a holdout accuracy of 0.9912. Given 95% confidence interval, split conformal prediction resulted in a definitive single-class prediction for 99.1% of instances, suggesting that uncertain cases constitute only a negligible number and require only an escalation in clinical practice. The results of decision curve analysis have shown that there is always net clinical benefit across all clinically relevant probability thresholds. Finally, multi-task analysis has shown that cytological morphology is useful for the purposes of breast cancer diagnosis but not prognosis.