Human-Centered Visual Segmentation

Human-centered visual segmentation incorporates human input—provided as visual and language prompts—to guide models in segmenting objects of interest across diverse visual modalities. This thesis addresses three primary challenges in this domain: minimizing human effort, optimizing computational efficiency, and unifying segmentation across diverse visual modalities. This thesis focuses on three prevalent modalities: 2D images, videos, and 3D medical volumes. For 2D images, we first introduce a click imitation mechanism that generates pseudo clicks to reduce human effort. We then show that scaling up the backbone with large-scale pre-trained vision transformers (ViTs) can achieve state-of-the-art performance. Finally, we present a next-generation interactive segmentation approach with low latency, high quality, and diverse prompts. For videos, we describe the first light memory network utilizing linear attention for video object segmentation. This innovative approach enables efficient segmentation of high-resolution videos, opening the door for the development of high-resolution video foundation models. For 3D medical volumes, we observe that video-trained mask propagation networks can be directly applied to this modality. However, their performance remains unsatisfactory, especially in cases with complex spatial structures. To address this, we propose a cycle-consistency loss that regularizes the training process, significantly improving the performance. Finally, we propose a promising future direction: unifying promptable segmentation for 2D images, videos, and 3D medical volumes via sequence-to-sequence modeling. Our initial results highlight the potential of a unified architecture to handle these modalities effectively.