Evaluating Coding Proficiency of Large Language Models: An Investigation Through Machine Learning Problems

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, but their effectiveness in coding workflows, particularly in machine learning (ML), requires deeper evaluation. This paper investigates the coding proficiency of LLMs such as GPT and Gemini by benchmarking their performance on three ML problems: Titanic, MNIST, and Steel Defect. These problems were chosen to encompass a range of challenges, including handling missing data, feature engineering, deep learning architectures, and multi-label classification. Using systematic prompts, we evaluated the LLMs’ abilities in data preprocessing, hyperparameter tuning, and classifier generation, comparing their outputs with those of human developers and AutoML frameworks. Experimental results indicate that the human developer outperformed untuned LLMs in data preprocessing, maintaining a 3–5% accuracy advantage across datasets. However, GPT’s hyperparameter tuning improved model performance by up to 6.3% in Titanic and 3.33% in Steel Defect, surpassing human-tuned models in some cases. In contrast, Gemini exhibited only marginal tuning improvements (0.19–1.78%) and failed to compensate for preprocessing inefficiencies. These findings show that while LLMs can assist with ML coding tasks, they exhibit varying levels of efficiency depending on task complexity and preprocessing requirements. GPT demonstrated superior hyperparameter tuning capabilities, whereas both LLMs struggled with intuitive data preprocessing, particularly in feature selection and transformation. This study provides practical insights into the strengths and limitations of LLMs in ML workflows, offering guidance for their effective integration into real-world applications.