MultiSHAP: a Shapley-based framework for explaining cross-modal interactions in multimodal AI models

Abstract Multimodal AI models have achieved impressive performance in tasks that require integrating information from multiple modalities, such as vision and language. While existing model explanation methods, such as attention maps and Grad-CAM, offer insights into cross-modal relationships, they do not explicitly quantify fine-grained interactions between individual visual and textual elements and require access to model internals. Here we introduce MultiSHAP, an architecture-agnostic interpretability framework that leverages the Shapley Interaction Index to attribute multimodal outputs to pairwise interactions between fine-grained visual and textual elements, such as image patches and text tokens. MultiSHAP does not require access to model parameters, gradients, attention weights, or intermediate representations, but requires a query interface that permits separate perturbation of visual and textual inputs and returns a task-relevant scalar score, either directly or through a ranked output from which such a score can be deterministically constructed. Our approach provides: (1) instance-level explanations that characterize how specific visual and textual elements jointly affect a selected model output, and (2) dataset-level descriptive summaries of interaction patterns under a specified model, value function, feature partition, and masking operator. Experiments on multimodal benchmarks provide evidence that MultiSHAP captures behaviorally relevant input-level interactions, while pair-level evaluations provide partial evidence that the estimated patch–token correspondences contain information beyond unimodal saliency, and case studies illustrate its use for examining individual multimodal decisions. Although evaluated here on vision–language models, the underlying interaction formulation may in principle be extended to other modality combinations.