Knowledge Discovery in Databases Utilizing Large Language Models

Converting natural language questions into executable SQL commands, known as text-to-SQL parsing, has seen a surge in interest recently. Advanced models like GPT-4 and Claude-2 have demonstrated significant potential in this area. However, existing benchmarks such as Spider and Wiki SQL primarily focus on simple database schemas with limited data, highlighting a disconnect between academic research and practical applications. To bridge this gap, we introduce BIRD, a comprehensive benchmark for large-scale database text-to-SQL tasks. BIRD includes 12,751 text-to-SQL pairs across 95 databases, totaling 33.4 GB and covering 37 diverse professional domains. Our focus on real-world database values brings forth new challenges, such as dealing with noisy or incomplete data, aligning natural language questions with external knowledge in the database, and improving SQL efficiency for large datasets. Addressing these issues requires text-to-SQL models to go beyond traditional semantic parsing to better understand database content. Experimental findings emphasize the critical role of database values in generating accurate SQL queries for extensive data. Even state-of-the-art models like GPT-4 achieve only 54.89% accuracy in execution, far from the 92.96% human benchmark, underscoring ongoing challenges in the field. Additionally, our analysis of query efficiency provides insights into crafting optimized SQL queries for industrial use cases. We believe BIRD will play a crucial role in advancing real-world text-to-SQL applications. The leaderboard and source code can be accessed at BIRD Benchmark. As data complexity increases and the demand for rapid data retrieval grows, integrating AI models, especially Large Language Models (LLMs), to assist users in generating SQL queries from natural language is becoming increasingly important. This research outlines a system where LLMs effectively combine with metadata-driven approaches such as mapping connections, segment definitions, and business logic?to enable intuitive SQL query generation. The system's setup, benefits, and foundational patterns are demonstrated through test datasets and a Power BI presentation.