Sparse attention is all you need for pre-training on tabular data

Abstract In the world of data-driven decision-making, tabular data reigns supreme as the most prevalent and crucial format, especially in business contexts. However, data scarcity remains a recurring challenge. In this context, transfer learning has emerged as a potent solution. This study explores the untapped potential of transfer learning in the realm of tabular data analysis, with a focus on leveraging deep learning models—especially the Transformer model—that have garnered significant recognition. Our research investigates the intricacies of tabular data and illuminates the shortcomings of conventional attention mechanisms in the Transformer model when applied to such structured datasets. This highlights the pressing requirement need for specialized solutions tailored to tabular data. We introduce an innovative transfer learning method based on series of thoroughly designed experiments across diverse business domains. This approach harnesses Transformer-based models enhanced with optimized sparse attention mechanisms, offering a groundbreaking solution for tabular data analysis. Our findings reveal the remarkable effectiveness of enhancing the attention mechanism within the Transformer in transfer learning. Specifically, pre-training with sparse attention proves increasingly powerful as data volumes increase, resulting in superior performance on large datasets. Conversely, fine-tuning with full attention becomes more impactful when data availability decreases in downstream tasks, ensuring adaptability in situations with limited data. The empirical results presented in this study provide compelling evidence of the revolutionary potential of our approach. Our optimized sparse attention model emerges as a powerful tool for researchers and practitioners seeking highly effective solutions for tabular data tasks. As tabular data remain the backbone of business operations, our study promises to revolutionize data analysis in critical domains. This work bridges the gap between limited data availability and the requirement for effective analysis in business settings, marking a significant step forward in the field of tabular data analysis.