Integrating Reinforcement Learning Into Vision-Language Models for Object Localization

In the realm of Vision-Language Models (VLMs), current research has demonstrated remarkable performance in tasks such as visual question answering and image caption generation. However, challenges persist in fine-grained object-level perception, particularly in accurately localizing objects within complex scenes. Our study addresses this gap by in-novatively integrating reinforcement learning, specifically the Proximal Policy Optimization (PPO) algorithm, to enhance VLMs' capability in processing multimodal inputs for precise 3D coordinate prediction. We focus on refining the VLM's visual processing module using PPO for fine-tuning, optimizing its ability to fuse RGB-D image data with textual inputs. The optimized VLM generates high-dimensional vectors that serve as the state space for a new PPO model, trained to predict accurate 3D coordinates of target objects. This dual-stage approach not only improves the model's sensitivity to detail but also enhances its performance in high-precision object localization tasks, crucial for applications in robotics, autonomous vehicles, and aerial navigation. The effectiveness of the proposed method is validated through experiments on the GraspNet dataset, showcasing significant advancements in the accuracy of object localization within complex visual environments.