Robotmanipulation med vision-språkmodeller

This thesis investigates the application and fine-tuning of Vision-Language Models (VLMs) for robotic manipulation, focusing specifically on the Open- Source Vision-Language-Action Model (OpenVLA) framework. Although OpenVLA shows strong capabilities to generate robot actions through text and visual information, the model has two notable limitations: 1) there is a lack of a clear fine-tuning strategy and no systematic evaluation of the performance of fine-tuned models, and 2) the model only accepts text instructions and RGB information from the perspective of the primary camera, which is not sufficient for the robot to complete tasks efficiently and accurately. In this work, we first design a local robot setup and collect a task-specific dataset to fine-tune OpenVLA using Low-Rank Adaptation (LoRA) for efficient training. We then propose and evaluate several architectural enhancements to OpenVLA to incorporate additional modalities, including encoded depth images and real- time robot proprioception. Multiple fusion strategies are explored, including early fusion, cross-attention, and concatenation methods. Experiments on in- dataset tasks show that further training does not lead to better performance once the loss reaches a certain threshold, while under-training results in significant performance degradation. Similarly, experiments on novel tasks demonstrate that OpenVLA has limited generalisation capacity to out-of- dataset tasks. Among the various approaches explored, concatenating depth information along the sequence dimension, followed by a projection and direct input into the Large Language Model (LLM) backbone, yielded the most promising results. However, despite these improvements, performance remains limited. Achieving stronger generalisation and robustness will likely require training on a larger scale and leveraging specialised depth perception models to better encode geometric and contextual cues from the environment.