An Evaluation of Large Language Models for Code Optimization

Code optimization has traditionally been a manual and time-consuming process in which developers identify and correct coding inefficiencies and bad programming practices. Large Language Models (LLMs) bring a fresh approach to this labour, being trained with large amounts of source code, they can detect inefficient code fragments and propose improved ones. However, concerns about their reliability, consistency across different programming languages, and trust in AI-generated solutions have slowed their adoption as automatic performance optimization tools. In this work, the code optimization capabilities of four LLMs were tested with six real-world codes written by non-expert developers, aiming to improve their execution time while maintaining correctness. A fixed methodology and identical prompts are employed, along with well-known profilers to detect bottlenecks as a starting point for every code. The results show that ChatGPT-4 Turbo significantly outperforms the other models, achieving an average runtime improvement of 31.9×, demonstrating its effectiveness in automated code optimization.