Enhancing Human Action Recognition with Modular DVS-RGB fusion

In the field of human action recognition, RGB video and event data each have distinct advantages and inherent limitations. RGB data is susceptible to environmental noise, such as background clutter and varying lighting conditions. Event data, conversely, responds solely to changes in brightness, making it less effective at capturing static actions and prone to significant noise under fluctuating illumination. Traditional multimodal fusion techniques, particularly Late fusion, are often limited to merely combining the final outputs from independently processed modalities, thereby limiting the exploitation of complementary feature interactions.To address these challenges and leverage the strengths while mitigating the weaknesses of both modalities, we propose a novel modular neural network architecture. Our approach focuses on enhancing classification performance by integrating an additional modular component—consisting of an RGB-to-Event generator and a shared embedding layer—without altering the underlying human action classification model itself. The proposed architecture uses a multitask learning mechanism to simultaneously perform RGB-to-Event generation (as an auxiliary task) and human action classification (as the primary task). Critically, the RGB-to-Event generation process reduces the RGB data's sensitivity to environmental noise and equips RGB features with the dynamic change detection capabilities characteristic of event data. This synergistic process enables the learning of robust intermediate representations capable of effectively handling both static and dynamic actions.