Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

Large language and multimodal models have shown remarkable success on various benchmarks focused on specific skills such as generalpurpose programming, math word problemsolving, and visual question answering.However, it is unclear how well these models perform on tasks that require a combination of these skills.In this paper, we curate a novel program synthesis benchmark based on the realworld tasks in the XLogoOnline visual programming environment.Each task requires a combination of different skills such as spatial planning, basic programming, and logical reasoning.Our evaluation shows that current stateof-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success rates, respectively.Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80, 000 tasks.Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution, through which a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models.Finally, we provide an in-depth failure analysis to understand the limitations of different models.We will publicly release the benchmark for future research on program synthesis in visual programming.