Transformer-Driven Semantic-Spatial Adaptive Fusion Representation for Object-Goal Navigation

Visual object-goal navigation requires an agent to make decisions to search for specified target objects within an unknown environment. While learning-based approaches have achieved progress, they still face two limitations: (1) visual representation: the lack of a low-dimensional representation that balances object priors with current observations, and the absence of effective modeling of scene spatial structures, which restricts the agent’s ability to perceive its surroundings. (2) policy learning: existing reward functions often fail to consider the visual essence of object goal-driven tasks, leading to inefficient learning and suboptimal performance. To address these issues, this paper introduces a semantic-spatial fusion representation framework that incorporates both object semantics and the intrinsic spatial structure of the scene. Specifically, a new object context matrix captures semantic relationships between objects while providing distinct low-dimensional representations for different observations, and an episode memory graph is also constructed and dynamically updated based on observation similarity and geodesic distance to represent the spatial structure of the agent’s environment in real-time. Then, a transformer-based adaptive multimodal feature fusion module is proposed to integrate these dual representations. Moreover, a sparse reward function is designed based on the target’s bounding box to guide the agent to learn correctly. The proposed method is the first to be evaluated on both the simulation platform AI2-THOR and the real-world dataset AVD, demonstrating its generalization in unseen environments. The method is also deployed on a physical mobile robot and tested in real-world scenarios, further validating its practical effectiveness.