Bottlenecks of Vision Language Models in Image to PDDL State Extraction

The advent of Large, Pre-trained, Vision or Language Models (LLM, VLM, etc.) has lead to their wide use in multiple robotic applications. To better understand how they can be leveraged in robotics applications in generating a precise description of an environment from an image, we evaluate the generation of Natural Language (NL) descriptions from images by comparing different prompts, image-to-PDDL data and state-of-the-art Vision-Language Models (VLMs). Results reveal specific strengths and weaknesses of individual VLMs and a strong reliance on image complexity in terms of object shape, cardinality and position, when generating NL descriptions.