Evaluating Column Type Annotation Models and Benchmarks

Column Type Annotation (CTA) is the process of assigning semantic types to the columns of a table. Models and approaches for CTA are typically evaluated on distinct datasets, making benchmarking and comparisons not trivial. In this paper, we identify major issues in commonly-used datasets for CTA and explore their impact on the evaluation process and reported results. Drawing from our analysis, we propose changes towards more robust and unbiased benchmarking practices.