Do foundation models work for geospatial tabular data? An investigation of TabPFN and a proposed enhancement based on geospatial sparse attention

Large-scale pretrained foundation models have transformed the machine learning paradigm. However, their applicability to Geospatial Tabular Data (GTD), particularly for regression tasks, remains poorly understood. This study systematically investigated TabPFN, a recent tabular foundation model, across synthetic data-generating processes and real-world geospatial regression tasks. We showed that while TabPFN generally outperforms state-of-the-art baselines, its performance degrades on large datasets and under strong, localised spatial dependence. To address these limitations, we proposed Geospatial Sparse Attention (GSA), a spatially informed inference strategy that injects geospatial inductive bias into the off-the-shelf TabPFN framework. The resulting model, TabPFN-GSA, prunes redundant attention calculations to better balance local and global spatial effects while improving scalability to large datasets. Empirical results showed that TabPFN-GSA delivers more accurate and robust predictions, particularly for large-scale GTD. Theoretically, this work advances our understanding of the strengths and limits of tabular foundation models in spatial contexts. Methodologically, it offers TabPFN-GSA, a principled, spatially explicit bridge between classical spatial modelling and modern foundation models.