Learn From Mistakes: Guidance on Zero-shot Conversational Text-to-SQL

Large language models (LLMs) possess powerful contextual comprehension capabilities and have demonstrated remarkable success in conversational tasks. However, existing works that apply LLMs to conversational text-to-SQL task have the problem of repetitive mistakes, which results in the failure to bring out the performance of LLMs. In this paper, we propose a novel approach that provides guidance through learning from mistakes. Specifically, the guidance offered by our approach includes tailored suggestions, corrective feedback, and personalized strategies aimed at improving learning outcomes. Furthermore, we employ chain-of-thought (CoT) to utilize guidance that is not suitable directly as prompts. Our method rigorously analyzes actual errors and strategizes on how to utilize the derived guidance effectively. Experimental results demonstrate that our approach improves the state-of-the-art (SOTA) performance metrics, increasing QEX performance from 66.3% to 70.9% (an absolute improvement of 4.6%) and IEX performance from 37.4% to 45.1% (an absolute improvement of 7.7%) on the CoSQL dataset.