LangSeg-BC: Language-Grounded Explainability for Breast Cancer Histopathology Segmentation

Current semantic segmentation models are able to generate highly accurate pixel-wise masks of tissues in images of histopathological samples. However, they do not provide any form of explanation, in the form that a pathologist can read, of the evidence on which a mask was determined. Attributions from post-hoc attribution methods such as saliency, feature relevance and sensory severity, highlight internal activations of the network but do not provide any form of language-based explanation of the model’s reasoning for a given mask. This work aims to bridge the gap between a segmentation mask and language-based explanation of that mask, and is different from generating a report about an image, where the image is the input and the segmentation is completely independent. To this end, we propose LangSeg-BC, a model that is composed of a Swin-Transformer U-Net, a Cross-Modal Attention Bridge (CMAB) and a Pathology Report Decoder (RRD). The text generated by the RRD is conditioned on the predicted masks rather than the input image. This dependence is implemented explicitly through a Mask Embedding Module, which can be removed to test how much the report relies on the mask. We evaluate LangSeg-BC on the Breast Cancer Semantic Segmentation (BCSS) dataset, which contains 151 regions of interest annotated at the pixel level into five tissue classes, and compare it with eight segmentation baselines and one captioning baseline, all reimplemented and trained with the same preprocessing, data split, and training schedule. LangSeg-BC achieves an mIoU of 0.843 and a Dice similarity coefficient (DSC) of 0.871 for segmentation, and a BLEU-4 score of 0.612 and a Tissue-Description Alignment (TDA) score of 90.5% for report generation. When the mask embedding is removed, segmentation is almost unchanged (mIoU 0.841), whereas BLEU-4 falls from 0.612 to 0.544 and TDA falls by 19.0 percentage points. This indicates that the report content depends on the predicted mask itself and not only on image features shared upstream. All results come from a single training run on a fixed test split, TDA measures agreement with visible morphology rather than diagnostic accuracy, and no clinical validation has been carried out.