When the Metric Rewards Giving Up: Nine Prompting Techniques Against a Plain Instruction on Three Deployed LLM Stacks
Nine widely cited prompting techniques were tested against a plain instruction on two Russian-language tasks and three LLM stacks as they are actually deployed: gemma3:12b-it-qat over a bare Ollama API, gpt-5.6-sol through the Codex CLI, and claude haiku through the Claude Code CLI. 7937 valid responses. On intent classification against a hand-labelled gold standard of production data, no technique beat the plain instruction on any stack. The highest accuracy in the study, 0.787, belongs to the plain instruction on claude haiku, and the only two Holm-corrected significant results are losses: chain-of-thought at 0.500 (p = 0.001) and chain of draft at 0.527 (p = 0.021), both on the weakest stack. The spread across techniques within a stack — 0.047, 0.060 and 0.213 — tracks stack capability rather than technique choice. On constrained title generation, the same technique moved the score in opposite directions: chain-of-thought scored +0.30 on gpt-5.6-sol and −0.30 on gemma3-12b, both surviving correction for multiple comparisons. The paper then holds the responses fixed and varies only the rule that scores them. Under a lenient inclusion check, chain-of-thought leads the field at 0.940 against 0.760 for the plain instruction; requiring literal inclusion drops it to 0.720; adding one condition — the title must differ from the query — drops it to 0.220 while the plain instruction holds 0.540. The mechanism is a degenerate answer: echoing the query back satisfies length, inclusion, no-quotes and no-period at once, and chain-of-thought produced it in 25 of 50 cases, chain of draft in 33 of 50. A formal metric assembled from independent checks admits answers that pass every check without performing the task, and the technique that best exploits the metric wins the table. Files in this record: prompt-techniques-preprint.pdf — the preprint, 11 pages, 4 figures, 6 tables. prompt-methodology-data.zip — raw model responses (JSON Lines, one object per call), the harness that produced them, the hand-labelled gold standard of 1810 queries, a catalogue of 54 techniques with sources, and the analysis scripts. Every number, table and figure in the paper is regenerated from the raw responses by these scripts.
Authors
- Stanislav Kirichenko (ORCID: https://orcid.org/0009-0001-2914-9541)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-08-28
- DOI
- https://doi.org/10.5281/zenodo.22144455
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint