Table Talk

Recent advancements in Natural Language Processing and Deep Learning have significantly transformed question-answering (Q/A) systems. However, Tabular Q/A remains a persistent challenge due to the prevalence of structured data in tabular formats. The hardware needs to deploy transformer-based language models in a production environment scale linearly with the size and number of tables. The methods based on the Retrieval Augmented Generation (RAG) techniques have improved computational efficiency in designing Q/A systems. However, choosing an appropriate value of "K" to retrieve the "K" number of most relevant records based on the user queries in a real-time production environment remains a notable challenge among all RAG-based approaches. This paper proposes a novel solution that enhances traditional RAG using Bi-Encoders and the Silhouette Method. It targets Non-Semantic Free-Form Closed-Domain Tabular Q/A, achieving an average precision of 63% and an average recall of 55%.