Open-Vocabulary Semantic Segmentation with Uncertainty Alignment for Robotic Scene Understanding of Indoor Building Spaces

Abstract Society is steadily embracing the idea of coinhabitant service robots in the built environment for assistance with household tasks and chores. In particular, the growing number of older adults, individuals with disabilities, and people requiring daily assistance has further created an urgent need for intelligent coinhabitant robots capable of supporting mobility, independence, and safety within physical living spaces. The growing complexity of modern built environments and the increasing prevalence of coinhabitant robots have underscored the need for advanced computational methods to support analysis, monitoring, and management. To provide users with meaningful assistance and clear scene understanding in such settings, robots must be able to perceive, interpret, and reason about their surroundings in a way that aligns with human spatial understanding. Place segmentation, which delineates spatial regions such as rooms or functional areas, together with semantic recognition, which assigns meaningful labels to these regions, forms the basis for accurate scene understanding in building-scale environments. Existing deep learning approaches have largely relied on closed-vocabulary detection systems, which are limited to a predefined set of categories of rooms and therefore struggle to interpret diverse or naturalistic descriptions of spaces. Furthermore, most current methods neglect the uncertainty inherent in scene recognition—for example, when rooms have overlapping functions (e.g., kitchen–dining areas) or when people use linguistically similar descriptions to refer to distinct spaces—thereby reducing their effectiveness in ambiguous or complex environments. To address these challenges, we propose an open-vocabulary semantic segmentation and classification pipeline that leverages vision–language models (VLMs) and large language models. Our framework follows a “segment–detect–select” paradigm for open-vocabulary scene understanding, consisting of three modules: (1) segmentation of distinct rooms and regions in the scene; (2) VLM-based similarity scoring between descriptive queries and segmented regions; and (3) adaptive conformal prediction to select among candidate regions while explicitly accounting for uncertainty. Evaluation results on the large-scale Matterport3D dataset, which comprises diverse photorealistic indoor environments, show that the proposed framework outperforms state-of-the-art open-vocabulary methods in scene segmentation [by 30% in AP50 and 5% in mean intersection over union (mIOU)], classification (by 200% in F1 score and 263% in mean average precision), and selection (by 50% in room mIoU). These advances are significant for built environments, as improved scene understanding directly supports safer, more adaptive, and more intuitive coinhabitant assistive agents for humans in complex indoor spaces.