CMAS: A Cross-Modal Attribution Score for Modality-Level Explanation Evaluation in Audio-Visual Deepfake Detection

Multimodal deepfake detectors combine visual and audio information, but their evaluation usually focuses on whether the final REAL or FAKE prediction is correct. That leaves a separate question: when a detector explains a fake clip, does the explanation point to the modality that was actually manipulated? FakeAVCeleb provides a useful setting for this question because its clips include labels for video-only, audio-only, and both-modality manipulations. I introduce the Cross-Modal Attribution Score (CMAS), a simple metric that compares a two-element visual and audio attribution vector with the corresponding manipulated-modality label using cosine similarity. I evaluate CMAS with a multimodal detector built from an EfficientNet-B0 visual encoder, a Wav2Vec2-base audio encoder, and a cross-attention fusion module. On the final 3,103-clip test split, the multimodal model reaches 99.55% accuracy and 99.77% F1. The visual-only and audio-only baselines reach 97.23% and 84.56% accuracy, with F1 scores of 98.56% and 91.49%, respectively. For the reported CMAS analysis, modality-ablation attribution has a mean score of 0.9997 and Integrated Gradients has a mean score of 0.9466 on 125 FAKE samples with defined manipulated-modality labels. I also document a calibration problem found during development: balanced sampling was used together with inverse-frequency loss weighting, which placed the decision threshold poorly for the initial models. Removing the extra loss weighting and retraining the audio-only and multimodal models produced better-calibrated default predictions. The results are presented with the limitations of the dataset, the final split, and the CMAS sample size stated explicitly.