deki: A MULTIMODAL IMAGE-TO-CODE TRANSFORMATION AND VISION-BASED GUI AGENTS

Accurately understanding graphical user interface (GUI) elements from screenshots remains a major challenge for current large language models (LLMs) and visual language models (VLMs).This paper presents deki, a multimodal system designed to analyze UI screenshots and produce rich structured descriptions of their constituent elements, including classes, precise bounding boxes, text content, and visual attributes.The main contribution of deki lies in its dual ability to first significantly improve the accuracy and fidelity of image-tocode generation by providing LLMs with explicit spatial and semantic information, and second, to enable the development of robust vision-based GUI agents capable of accurate interaction.Preliminary results show that augmenting LLM with deki results improves UI layouts in code generation tasks, as measured by the structural similarity index (SSIM), and facilitates successful task automation by GUI agents, achieving 100% success in a pilot study of 10 single-step tasks, compared to 20% for the baseline without deki.deki has the potential to have a broad impact on multimodal AI applications, particularly in areas that require fine-grained understanding and interaction with graphical interfaces, such as automated software engineering, accessibility tools, and advanced human-computer interaction.