Does a Security Skill Beat a One-Line Reminder? A Blinded, Pre-Registered Test of Agent Skill Files for Safer AI-Generated Code

AI coding agents often build functional apps and websites, but implicitly leave out security features that were not requested. Agent skills (packaged instruction files) are often used to combat this, but we found little evidence comparing them with simply adding a line of security advice to the prompt. We built a security skill and tested it against a one-line reminder, no guidance, and a public skill on Codex CLI. This was done with pre-registered rules, blinded code review and a feature-fidelity check. On a strong model (gpt-6.1-sol), the reminder was sufficient to eliminate all critical failures (runs with at least one critical security failure) on the original seven prompts (0/35 runs). On a weaker model (gpt-5.6-luna), the skill was not detectably better than the reminder in two rounds on fresh variants, or on ten probes mapped to the OWASP Top 10, and the public skill did no better than the reminder. A final 400-run phase added two extra arms: a security checklist, and a checklist-length placebo with no security content. On the three prompts where no-guidance runs sometimes passed, critical failures fell from 23/30 to between 6/30 and 9/30 with the reminder, checklist or skill. The placebo did not help (22/30). Across all eight prompts the skill showed no detectable advantage over the reminder (27/80 against 23/80; difference +5 points in failure rate, 95% interval −9 to +19) and used 1.7 times the tokens of no guidance. The checklist-placebo contrast suggests that security content, rather than instruction length, carries most of the benefit. Preprint; submitted to AGENT 2027 at ICSE 2027, not yet peer reviewed. Data, code and criteria: https://doi.org/10.5281/zenodo.23498723