Towards Complex Table Question Answering Over Tabular Data Lakes (Extended Version)

Abstract Natural Language Interfaces for Databases (NLIDBs) offer an interesting alternative to SQL since they empower non-experts to query data. However, they require this data to be integrated into a database schema, causing high data engineering and integration overheads. As such, Open Table Question Answering (OTQA) is promising since it allows directly querying tables in data lakes without first incorporating them into a relational schema. Many recent OTQA approaches combine retrieval-augmented generation with Large Language Models (LLMs), where relevant tables are first retrieved from a data lake and then used as input to an LLM to answer the user query. In this paper, we systematically analyze how LLMs paired with table retrievers can answer queries over private tabular data lakes. We find that the answer generation often fails because the retrieval step does not provide the required tabular context. To overcome this issue, we propose a novel LLM-based retrieval approach called Zoom retrieval, which effectively boosts retrieval accuracies and thereby improves question answering results. Nevertheless, LLMs often still fail to answer even simple extraction queries, let alone aggregates, thus remaining far from the rich querying capabilities that NLIDBs offer today. Therefore, future work should focus on improving the query execution capabilities of LLMs to enable complex question answering over tabular data lakes.