Efficient Multi-Head Attention for Human-Object Interaction Recognition and Video Scene Graph Generation

Comprehending human-object interactions in video sequences is one of the fundamental problems in visual classification and an essential step toward more detailed scene understanding. Scene graph generation (SGG) has proven to be a powerful tool for modeling these relationships by representing interactions as graph structures. However, while capturing temporal relationships, dynamic video scene graph generation methods often introduce significant computational complexity and struggle with long-tail distributions, making rare interactions challenging to detect. In this paper, we propose a static approach for scene graph generation that leverages multi-head attention mechanisms. Each attention head specializes in different types of relationships, such as spatial, contact, and attention-based interactions. The proposed method offers a more efficient and accurate way to recognize multi-type relationships by refining visual features through contextual information and focusing on the most relevant aspects of each interaction. This paper concludes with a comparison of the proposed method against both static and dynamic leading approaches for video scene graph generation, highlighting its effectiveness on the Action Genome dataset.