Utilizing Past User Feedback for More Accurate Text-to-SQL

In the classical problem formulation of Text-to-SQL in academia, each question is translated independently of the others into SQL. This differs from the setting in practice, where questions enter the Text-to-SQL system in sequence. Thus, for all but the first few questions, a translation history is available that contains past questions and how they were translated by the system. So far, it has not been sufficiently explored how Text-to-SQL systems can make use of the translation history to improve future translations. Another crucial difference from the academic setting is that in practice, users generally have a conversation with the Text-to-SQL system. More concretely, if the initial translation contains a mistake, the user can follow up with feedback messages to allow the system to fix the mistake. In this case, it might be helpful to remember these user feedback messages to avoid repeating past mistakes. Thus, in this paper, we explore how a history of such past conversations between users and the Text-to-SQL system can be used to make future Text-to-SQL translations more accurate. We explore several approaches for extracting relevant experiences and insights from this conversation history and show in an evaluation that utilizing them can improve translation accuracy by up to 14.9%.