HAETAE: In-domain Table Pretraining with Header Anchoring

Understanding structured table data with language models is crucial for various downstream tasks in information retrieval.However, transformer-based table embedding models struggle to consistently represent headers across varying entity contexts.This inconsistency undermines the generalizability of embeddings across in-domain tables that share universal semantics.To address this gap, we propose a novel pretraining method for in-domain tables, HAETAE, that explicitly separates header embeddings from contextual entity embeddings.Our method introduces a dedicated header encoder and learnable alignment mechanisms, built upon header-aware serialization.Experimental results demonstrate that HAETAE enhances generalization and stability in predicting headers and values of in-domain tables, achieving higher accuracy than baselines while showing the notable potential of knowledge transfer in cross-domain tables.The source code of HAETAE is available at https://github.com/woojoonjung/HAETAE.