Understanding and exploiting fundamental modality advantage in frame and event hybrid visual data

Abstract Visual perception systems increasingly rely on hybrid sensing modalities, including RGB and event cameras, to operate robustly under complex and extreme environmental conditions. However, what fundamentally determines the respective advantages of RGB frames and dynamic events remains unclear. To this end, we introduce MAD-Drone (Modality Advantage Dataset for Drone Perception), a multi-condition visual dataset built in a physics-based simulation framework with a high-fidelity event camera model to provide explicit modality-aligned definitions. This enables to map how modality advantages vary across data conditions and introduce several interpretable modality-aware metrics to explain these variations within a unified space. Given systematic understanding, we finally design a lightweight plug-in module alongside simple modality-specific preprocessing strategies to translate data-driven insights into modality usage. Over multiple real-world datasets, including DSEC, PKU-DAVIS-SOD, NeRDD and our physically-collected MAD-Drone, spanning different object categories, consistent performance improvements are demonstrated. Our findings reveal previously underappreciated patterns: modality advantage is not determined by coarse-grained condition labels or any monotonic superiority of event-based sensing, but emerges from coupled reliability conditions involving the temporal variation, illumination statistics, and foreground-background information density. Ultimately, this work provides a data-centric foundation for understanding, explaining, and exploiting fundamental modality advantages in frame and event hybrid visual data.