ORSA-T: Multi-View Object-Centric Scene Representation Learning with Slot Attention and Transformer

Understanding a scene from multiple, potentially partial views and decomposing it into objects is foundational for human perception and intelligence. Current multi-view object-centric scene representation learning models that use partial views analyze all views at once. This differs from the way humans process visual information and is not compatible with reinforcement learning, where an agent learns about its environment through actions, such as moving to change the viewpoint. In this paper, we propose ORSA-T (Object-centric scene Representation learning with Slot Attention and Transformer), which combines Implicit Slot Attention with an aggregation of previous views by a Transformer and improves the scene representation iteratively based on a sequence of images annotated with viewpoints. The Transformer uses all previous representations and the current update to aggregate scene information, which makes ORSA-T remember objects better and learn more effectively when applied to partial views. In our experiments, ORSA-T predicts and segments images from a new viewpoint better than MulMON, the current SOTA, and ORSA without aggregation connections and Transformer. As ORSA-T learns iteratively to improve its scene representation, it is suitable for use in reinforcement learning.