What Do Resume–Job Matching Benchmarks Actually Measure? Keywords, Embeddings and LLMs on Public Data

Abstract Automated resume–job matching ranks job candidates, but most studies use private data. We examine what is, to our knowledge, the only public resume–job benchmark with graded fit labels, comparing keyword methods, sentence embeddings (zero-shot and fine-tuned) and a small open LLM judge (Qwen3-4B). Across 294 jobs, zero-shot embeddings perform slightly but significantly better than TF-IDF (NDCG@10 0.739 vs 0.706). Fine-tuning improves the results, but 430 of 434 test resumes also appear in training. In cross-validation, seeing them explains most of the gain even at equal training size, yet a model trained without them still beats zero-shot methods. The LLM judge does not perform significantly better than embeddings and is about 850 times slower. Good vs Potential Fit is largely set by the job and, within a job, follows the resume more than the match, even when estimated from unrelated jobs. Against the dataset labels, no job-aware method separates these levels significantly better than chance, while a baseline that ignores the job reaches an AUC of 0.727. In a blind relabelling of 150 pairs, the author agrees weakly with the dataset (κ = 0.324 for fit vs no fit); on 40 pairs, a second annotator agrees with the dataset (κ = 0.319) more than with the author (κ = 0.093). Finally, adding markdown without changing any words increases a resume's E5 score by 0.91 standard deviations, regardless of fit. We therefore recommend unseen-resume splits, reliable labels, resume-only baselines, and markdown removal before embedding. We release our code, splits and labels.