Exploring the performance of large language models in novel command-based tasks

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language tasks, yet their performance in structured command-based environments remains understudied. This paper evaluates seven leading LLMs—GPT-4o mini, GPT-o1, Deepseek-V3, Deepseek-r1, Claude 3.5 Sonnet, Gemini 2.0 Flash, and Llama 3—on two novel command-based tasks: a pixel painting application requiring precise syntax adherence and a maze navigation task demanding spatial reasoning. Our comprehensive evaluation framework assessed syntax adherence, visual accuracy, and command efficiency across multiple iterations. Results revealed significant performance disparities between models, with notable variations in their ability to interpret commands accurately and reason spatially. Only one model successfully completed all maze navigation tasks, while performance on pixel painting revealed a clear hierarchy in both accuracy and efficiency. This stark performance gap, particularly in spatial reasoning, highlight limitations in most current LLMs’ ability to execute sequential, rule-based commands in structured environments. Our findings contribute to understanding the boundaries of LLM capabilities in command interpretation and procedural reasoning, with implications for applications in robotic control, programming assistance, and interface automation. We propose several directions for future research, including expanded task diversity, architectural analysis, and specialized fine-tuning approaches.