RAGSQL: Context Retrieval Evaluation on Augmenting Text-to-SQL Prompts

Text-to-SQL area evolves rapidly in the era of large language models (LLMs) as organizations look for adapting the technology. A gained popularity method to improve LLM outputs is through giving relevant context about the database and organizational data as few-shot prompting. Recent research showcases that better results in text-to-SQL tasks can be achieved in a Retrieval Augmented Generation (RAG) setting by augmenting the prompt context in dynamic fashion during the LLM inference. In this paper we design such experiment by adapting data from BIRD benchmark, on more than 10 thousand examples and evaluate existing embedding models, both closed and open source, for such tasks reporting the comparison results. We then fine-tune an open-source sentence transformer model that outperforms the state-of-the-art (SOTA) embedding models on both question-to-evidence and question-to-SQL retrieval experiments.