Probing Computational Video Representations with MAVIC: A Multimodal Assessment Vocabulary for Interpretability and Coverage

When researchers analyze thousands of videos, their choice of computational representation shapes which similarities and differences they can detect. However, standard model benchmarks do not show whether a representation captures the visual, audio, and linguistic properties relevant to a particular study. We present MAVIC, the Multimodal Assessment Vocabulary for Interpretability and Coverage, which assesses video representations against 133 observable features grounded in communication research. These comprise 86 visual, 13 audio, and 34 linguistic features. We apply MAVIC to 20,180 videos sampled from 1,009 donated YouTube watch histories and compare three representation pipelines using images, audiovisual content, or metadata. For each pipeline, linear probes predict every MAVIC feature from its representation, producing an interpretable coverage profile. The three profiles differ markedly: pipelines tend to capture features from the modalities they receive most strongly, but they also encode indirect associations across modalities. Some features that vary across demographic groups’ observed video exposure are captured only weakly, indicating possible substantive blind spots. MAVIC therefore enables researchers to evaluate whether a representation preserves the distinctions their research question requires, rather than selecting pipelines based on general performance alone.