Investigating LLM Capabilities in the Identification of Compilation Errors in Configurable Systems

Compilation is an important process in developing configurable systems, such as Linux. However, identifying compilation errors in configurable systems is not straightforward because traditional compilers are not variability-aware. Previous approaches that detect some of these compilation errors often rely on advanced techniques that require significant effort from programmers. This study evaluates the efficacy of Large Language Models (LLMs), specifically CHATGPT4, GEMINI ADVANCED 1.5, LE CHAT MISTRAL, and LLAMA 3, in identifying compilation errors in configurable systems. We evaluate them in 30 small configurable systems in C, covering 17 different types of compilation errors. CHATGPT4 successfully identified 28 out of 30 compilation errors. LE CHAT MISTRAL, LLAMA 3 and GEMINI ADVANCED 1.5 detected 24, 20, and 16 errors, respectively. LLMs have shown potential in assisting developers in identifying compilation errors in configurable systems.