Evaluating ChatGPT for Text-To-SQL: An Empirical Approach

Since the existence of database systems, the querying of data managed by them has been one of the most important and challenging task for their users. For this reason, from the very beginning, there were efforts to allow users to query data using natural language (in most cases English). In such a case, the text has to be processed by a system which generates the corresponding query. In this article we use an empirical approach to evaluate the use of ChatGTP as a representative of Large Language Models (LLMs) for creating and querying relational databases. Our results show that two groups of queries cause significant problems for the system. First, if the system has to „think outside the box", it generates, in several cases, an erroneous query. Second, if a query, which has to be generated is a complex one or its generation demands the use of rarely applied SQL constructs, the corresponding SELECT statement is sometimes faulty.