Evaluating ESCO skill classifiers

ESCO skill classifiers, which scan job ads to identify skills at the finest level of the taxonomy, are widely used in international statistical projects and by national employment agencies. However, to our knowledge, no systematic evaluation of these classifiers has been conducted across large sets (i.e., thousands) of ESCO skills. We introduce a method for evaluating ESCO skill classifiers, addressing two key challenges: the large number of skills (up to around 14 000) and severe class imbalance. Our approach relies on matrix sampling of skills and job ads, with clustering and stratification, and uses bootstrapping to estimate standard errors for classifier comparisons. We apply the method to three classifiers using a sample of Luxembourgish IT and finance job ads. Our results indicate that the classifiers achieve acceptable accuracy despite low recall. Notably, the text matching classifier performs competitively, even against methods based on large language models.