Frequency-adaptive prototype learning for long-tail scene graph generation

Abstract Scene graph generation aims to detect visual objects and their relationships in images, providing structured representations for visual scene understanding. However, it faces fundamental challenges from extreme long-tail distribution and semantic ambiguities between related predicates. The severe class imbalance—where head classes dominate with abundant samples while tail classes have scarce training instances—creates distinct learning dynamics that existing re-balancing methods fail to address by treating all classes uniformly. We propose Frequency-Adaptive Prototype Learning (FAPL), a unified framework that introduces three key innovations: (1) frequency-aware momentum scheduling that enables head prototypes to stabilize early as knowledge sources while tail prototypes maintain aggressive adaptation to maximize learning from limited samples; (2) pattern-aware supervision that learns semantic confusion patterns between related predicates and adaptively reduces penalties for plausible errors while maintaining full supervision for implausible ones; (3) curriculum learning integration that progressively shifts training emphasis from frequent to rare relationships as the model accumulates semantic understanding. These components form a mutually reinforcing system where early head learning establishes reliable knowledge bases, pattern-aware supervision reduces confusion noise, and curriculum scheduling maximizes tail exploitation. Extensive experiments on Visual Genome, GQA, and Open Images V6 datasets demonstrate that FAPL achieves state-of-the-art performance with substantial improvements over previous best methods: up to 30.3% on Visual Genome, 28.8% on GQA, and 19.9% on Open Images V6, while showing superior generalization capability on recognizing unseen relationship combinations.