Achieving self-healing code with LLMs

This study investigates LLMs for their potential in achieving self-healing capabilities in code. Ten LLMs were evaluated using a dataset of 76 problems sourced from Leetcode in two programming languages: C++ and Java. The programming problems are categorized into three difficulty levels: easy, medium, and hard. Leetcode classifies the programming problems into different difficulty levels. Each model was given a single attempt to correct the run-time errors, and their effectiveness was measured by their success rate, which calculates their ability to resolve errors and create functioning code. An experiment was conducted LLMs were prompted with Leetcode programming problems and their erroneous solutions, in order to evaluate these LLMs. In light of the findings considerable variations in model performance, with success rates ranging from 30.26% to 94.74%. The unified model approach achieved a moderate success rate of 65.53%. Integrating multiple models within a system can enhance reliability, when addressing coding problems. To ensure the solutions were corrected by the LLMs, manual testing through Leetcode’s hidden test cases provided the necessary evaluation framework. However, this method had disadvantages, such as being time-intensive and the possibility of introducing human error into the fray. The results demonstrate the feasibility of LLMs in achieving self-healing code. Even though current models are promising, some challenges remain, especially in integrating LLMs into software. Without a method to integrate the LLMs into actual software, their potential remains theoretical. Future research in this area should expand the dataset to include more problems and programming languages.