Keyword Stuffing and Hidden Text in CV Screening: A Benchmark of LLM-Based CV Matching and Keyword Search in an Open-Source Applicant Tracking System

Applicants who expect a keyword filter sometimes write for it: they hide the job's keywords in white or tiny text, pile them into a "keywords" line, paste the job advertisement into their CV, list skills their career does not show, or, with LLM-based screening in mind, add a hidden instruction asking the system to rank them first. We present a benchmark of 400 synthetic, full-length CVs for four jobs in which 128 candidates use one of eight such tricks, alongside honest controls (long genuine skills lists, and qualified people who pad their own real skills in white text) and an answer key frozen and hashed before any system processed the CVs. No trick makes anyone qualified in the key. We compare Formzed's large-language-model (LLM) CV matching with the resume search of OpenCATS 0.12.0, an open-source applicant tracking system, queried with the job descriptions' own terms and with the alternative terms a recruiter would add. OpenCATS with recruiter terms put 78 of the 128 cheaters (60.9%, 95% CI 52.3 to 69.0) on its shortlists; the LLM-based system put none (0%, 0 to 2.9). Across 163 qualified candidate-job pairs, the LLM-based system found 134 (82.2%, 75.6 to 87.3) with 1 wrong pick (precision 99.3%), against 97 (59.5%) with 97 wrong picks (precision 50.0%) for the best keyword configuration (exact McNemar test on qualified candidates, 48 vs. 11, p ≈ 1.2 × 10^-6). The LLM-based system set aside hidden text on all 64 CVs that used white, tiny, covered or instruction text, and on none of the honest CVs without such text; its check for copied and piled-up keywords caught all 32 pasted advertisements and keyword walls and flagged one honest CV. Its 29 misses were mostly strict judgements on leadership and ownership requirements. The system was developed on separate CV sets; the test set was generated afterwards and each system was scored on it once. We state the conflict of interest and the limits of a synthetic benchmark built around known tricks.