Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting

Abstract Ensuring trustworthiness and versatility in Open-World Visual Recognition is a core goal of Multimodal Large Language Models (MLLMs). While integrating segmentation is desirable, prevailing methods require invasive fine-tuning, which often leads to catastrophic forgetting of the model’s general-purpose capabilities. This trade-off between task-specific performance and general reliability fundamentally undermines the goal of building unified, trustworthy systems. We introduce LENS ( L everaging k E ypoi N ts for MLLMs’ S egmentation), a novel plug-and-play paradigm designed to equip MLLMs with pixel-level ability without compromising their foundational integrity. LENS attaches a lightweight head to a completely frozen MLLM, interpreting the intrinsic spatial cues embedded in attention maps. It extracts keypoints and translates them into prompt-compatible features for a mask decoder. Extensive experiments validate that LENS achieves segmentation performance competitive with retraining-based methods. Crucially, it largely preserves the MLLM’s open-world generalization, avoiding the degradation observed in fine-tuned counterparts. By decoupling segmentation from generation, LENS offers a robust and efficient path toward truly multi-talented and reliable vision models.