Quality assessment of GPT-3.5 and Gemini 1.0 Pro for SQL syntax

• GPT-3.5 has an accuracy of 87% for SQL syntax generation or correction, whereas the Gemini 1.0 Pro has 80%; • The quality of training data directly impacts the GPT-3.5’s and Gemini 1.0 Pro’s ability to generalize and provide accurate responses across various contexts; • GPT-3.5 and Gemini 1.0 Pro may automate certain aspects of programming; • Currently, GPT-3.5 and Gemini 1.0 Pro cannot replace human programmers, but they can augment their abilities and streamline specific tasks. Nowadays, GPT-3.5 and Gemini 1.0 Pro are employed for various tasks, both for personal and professional use, in multiple domains like education, economy, computer science, etc. Given the increase in users, knowing the quality level of these artificial intelligence (AI) tools is important. Thus, the paper presents a comparative analysis of syntax accuracy generated for SQL databases utilizing the services of GPT-3.5 and Gemini 1.0 Pro. Firstly, the algorithms for testing GPT-3.5 and Gemini 1.0 Pro were developed. Secondly, five types of tests, which implied 700 queries, were conducted by considering requirements with low and high degrees of difficulty. The tests focus on syntax-generated accuracy using an experimental (NorthWind) database, syntax-generated accuracy study using a user-made database, syntax correction accuracy, different responses to the same question on the same account, and different responses to the same question on other accounts. The accuracy obtained for all tests revealed that the GPT-3.5 service has a value of 87% for SQL syntax generation or correction, whereas the Gemini 1.0 Pro service has an accuracy of 80%. These results underscore the effectiveness of GPT-3.5 and Gemini 1.0 Pro in assisting with SQL syntax tasks, albeit with differing levels of precision. The findings highlight the significance of human supervision and validation in ensuring the correctness of AI-generated responses, particularly in database-related tasks. The results affect developers and database administrators when selecting appropriate tools for query requirements. For now, replacing programmers with GPT-3.5 and Gemini 1.0 Pro is impossible.