A Novel Depth Scene Classification with Vision Transformer and ResNet Model

Scene classification in indoor environments is a challenging task with numerous applications in robotics, augmented reality, and smart environments. In this paper we propose a novel scene classification model for indoor RGB-D images that combines unsupervised segmentation, vision transformers, convolutional neural networks, and conditional random fields. The pipeline commences with segmenting the input RGB-D data using DBSCAN, followed by feeding these segmented images into a Vision Transformer (VT) for feature extraction. The VT model incorporates two distinct branches: a feature extraction branch that captures fine-grained local patch representations, and a relation modeling branch that models long-range dependencies between patches through self-attention maps. These extracted features are subsequently fed into a ResNet-18 architecture for multiobject classification, providing a comprehensive understanding of the scene’s contents. Finally, the classified objects and the contextual information from the VT’s relation modeling branch are integrated into a Conditional Random Field (CRF) framework for scene classification.