Towards Complex Table Question Answering Over Tabular Data Lakes

Natural Language Interfaces for Databases (NLIDBs) are an interesting alternative to SQL since they empower non-experts to query data. However, NLIDB approaches require this data to first be integrated into a database schema, which causes high upfront data engineering and integration overheads. As such, Open Table Question Answering (OTQA) is promising, since it allows directly querying the data in data lakes without needing to incorporate it into a schema. Many recent OTQA approaches combine Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs), where relevant tables are first retrieved from a data lake and then used as input to an LLM to answer the user query. In this paper, we take the first systematic step for investigating how LLMs paired with table retrievers can answer queries over private tabular data lakes. As a main finding, we see that even when tuning several parameters of this approach, current LLMs still fail to answer queries that focus on the simple extraction of individual cell values, let alone aggregate queries. Thus, they are far from the rich querying capabilities that NLIDB approaches offer today. To solve this, we recommend promising future work to enable complex question answering over tabular data lakes.