A Regularization Method for Reducing Cross-Modal Blind Spots in Clinical CLIP Models

Abstract Multimodal clinical models built on contrastive learning can treat semantically distinct radiology report texts as interchangeable in their shared embedding space, creating blind spots invisible to standard evaluation metrics. Clinically distinct phrases such as “left” vs. “right lung opacity” may receive near-identical scores, propagating silently into down-stream pipelines. We extend the symmetry-matching framework from algebraic error-detection theory to cross-modal encoders, defining a cross-modal stabiliser and a measurable vulnerability index VGini on the cosine-similarity level. Across two medical CLIP architectures (BiomedCLIP and PubMedCLIP) and 3 783 perturbation pairs from 1 000 Indiana University chest radiograph reports, every clinically relevant perturbation family exhibits substantial baseline vulnerability (VGini = 0.46–0.64). A sim-level V -regularisation applied solely to the frozen-encoder projection head reduces VGini by 34–84% on real radiology reports (and by up to 95% on synthetic perturbation pairs), and increases downstream perturbation-detection AUC from 0.50–0.59 to 0.51–0.73 on BiomedCLIP, and from 0.49–0.56 to 0.52–0.60 on PubMedCLIP, without retraining either base encoder. SVD of per-family sensitivity matrices yields a closed-form orientation diagnostic: inter-family image-space angles of 2.4° (BiomedCLIP) and 0.9° (PubMedCLIP) between the negation and severity families formally establish the interference barrier that prevents any single projection head from simultaneously optimising both families. A sequential dual-head factorisation supports the m∗ > 1 transition [4]: on the synthetic ten-family corpus, dual-head training reduces VGini on all ten families, and sequential m=2 outperforms a single head on three of four PubMedCLIP families. The method adds one loss term and requires no base-model, data, or deployment changes.