Because: preregistered randomized tests of what makes a coding agent follow its instructions
People write instruction files for coding agents on folk advice: give a reason, write MUST, put the important rule first. I tested that advice on Claude Code in four preregistered randomized studies, with outcomes decided by code. Giving a reason came back inconclusive, and no wording factor survived correction on average. For skills, the largest effects came from the listing's character budget, which controls whether a description reaches the model. The numbers and their caveats are in the paper.
Authors
- Abhay Mettu (ORCID: https://orcid.org/0009-0001-2434-353X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-02
- DOI
- https://doi.org/10.5281/zenodo.23896685
- Primary Topic
- Software Engineering Research
- Type
- preprint