Gaze-Enhanced Multimodal Action Recognition for Real-Time Human-Robot Collaborative Disassembly

Human Action Recognition (HAR) is essential for safe<br/>and intelligent Human-Robot Collaboration (HRC) in Industry<br/>5.0 disassembly scenarios, where real-time performance and data<br/>efficiency are critical. Existing approaches either rely on large<br/>labelled datasets or fail to integrate complementary modalities<br/>effectively, particularly in environments where actions exhibit<br/>high inter-class similarity. We propose a multimodal few-shot<br/>HAR framework that combines RGB, depth, skeleton, gaze, and<br/>head pose through hierarchical cross-attention fusion, grounding<br/>operator intent in scene context. The combination of gaze and<br/>head pose with skeleton data represents a novel modality pairing<br/>for HAR in HRC, providing intent-aware recognition unexplored<br/>by prior architectures. A Mamba temporal block enables linear-<br/>complexity sequence modelling for real-time inference, and a<br/>prototypical network enables classification from as few as one to<br/>five labelled clips per action class. The framework is designed for<br/>validation in collaborative disassembly cells featuring industrial<br/>cobots, targeting actions with subtle distinctions dependent on<br/>manipulated objects.