Zero-shot human-object interaction detection via learning from the data itself

Human-Object Interaction (HOI) Detection aims to locate and recognize HOI instances in the form of in images or videos. However, the significant cost of manpower and resources for annotations poses challenges, particularly in addressing the long-tail or even zero-shot distribution problem. Additionally, the generation of visual features from fixed semantic information encounters the issue of a lack of diversity. In response of these challenges, we develop a novel Visual-Semantic and Multi-source Feature Generation (VSMG) network for zero-shot human-object interaction detection. Firstly, by combining the visual-semantic GAN, the model not only generates visual/semantic features from corresponding semantic/visual ones but also introduces the diverse unseen visual features during the training phase. Secondly, by employing a knowledge-aware relation graph, the model encodes the relations between objects, actions and interactions including both seen and unseen classes. Based on the relation weights in the relation graph, a dynamic multisource feature generation strategy is performed to generate diverse virtual visual features for unseen classes. Finally, experimental results on HICO-DET dataset validate the effectiveness of our proposed method, demonstrating improvements the detection performance of the trained HOI detector.