When the Metric Rewards Giving Up: Nine Prompting Techniques Against a Plain Instruction on Three Deployed LLM Stacks

Nine widely cited prompting techniques were tested against a plain instruction on two Russian-language tasks and three LLM stacks as they are actually deployed: gemma3:12b-it-qat over a bare Ollama API, gpt-5.6-sol through the Codex CLI, and claude haiku through the Claude Code CLI. 7937 valid responses. On intent classification against a hand-labelled gold standard of production data, no technique beat the plain instruction on any stack. The highest accuracy in the study, 0.787, belongs to the plain instruction on claude haiku, and the only two Holm-corrected significant results are losses: chain-of-thought at 0.500 (p = 0.001) and chain of draft at 0.527 (p = 0.021), both on the weakest stack. The spread across techniques within a stack — 0.047, 0.060 and 0.213 — tracks stack capability rather than technique choice. On constrained title generation, the same technique moved the score in opposite directions: chain-of-thought scored +0.30 on gpt-5.6-sol and −0.30 on gemma3-12b, both surviving correction for multiple comparisons. The paper then holds the responses fixed and varies only the rule that scores them. Under a lenient inclusion check, chain-of-thought leads the field at 0.940 against 0.760 for the plain instruction; requiring literal inclusion drops it to 0.720; adding one condition — the title must differ from the query — drops it to 0.220 while the plain instruction holds 0.540. The mechanism is a degenerate answer: echoing the query back satisfies length, inclusion, no-quotes and no-period at once, and chain-of-thought produced it in 25 of 50 cases, chain of draft in 33 of 50. A formal metric assembled from independent checks admits answers that pass every check without performing the task, and the technique that best exploits the metric wins the table. Files in this record: prompt-techniques-preprint.pdf — the preprint, 11 pages, 4 figures, 6 tables. prompt-methodology-data.zip — raw model responses (JSON Lines, one object per call), the harness that produced them, the hand-labelled gold standard of 1810 queries, a catalogue of 54 techniques with sources, and the analysis scripts. Every number, table and figure in the paper is regenerated from the raw responses by these scripts.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-08-28
DOI
https://doi.org/10.5281/zenodo.22144454
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When the Metric Rewards Giving Up: Nine Prompting Techniques Against a Plain Instruction on Three Deployed LLM Stacks

Stanislav Kirichenko
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

When the Metric Rewards Giving Up: Nine Prompting Techniques Against a Plain Instruction on Three Deployed LLM Stacks

Stanislav Kirichenko
preprint en

Abstract

Nine widely cited prompting techniques were tested against a plain instruction on two Russian-language tasks and three LLM stacks as they are actually deployed: gemma3:12b-it-qat over a bare Ollama API, gpt-5.6-sol through the Codex CLI, and claude haiku through the Claude Code CLI. 7937 valid responses. On intent classification against a hand-labelled gold standard of production data, no technique beat the plain instruction on any stack. The highest accuracy in the study, 0.787, belongs to the plain instruction on claude haiku, and the only two Holm-corrected significant results are losses: chain-of-thought at 0.500 (p = 0.001) and chain of draft at 0.527 (p = 0.021), both on the weakest stack. The spread across techniques within a stack — 0.047, 0.060 and 0.213 — tracks stack capability rather than technique choice. On constrained title generation, the same technique moved the score in opposite directions: chain-of-thought scored +0.30 on gpt-5.6-sol and −0.30 on gemma3-12b, both surviving correction for multiple comparisons. The paper then holds the responses fixed and varies only the rule that scores them. Under a lenient inclusion check, chain-of-thought leads the field at 0.940 against 0.760 for the plain instruction; requiring literal inclusion drops it to 0.720; adding one condition — the title must differ from the query — drops it to 0.220 while the plain instruction holds 0.540. The mechanism is a degenerate answer: echoing the query back satisfies length, inclusion, no-quotes and no-period at once, and chain-of-thought produced it in 25 of 50 cases, chain of draft in 33 of 50. A formal metric assembled from independent checks admits answers that pass every check without performing the task, and the technique that best exploits the metric wins the table. Files in this record: prompt-techniques-preprint.pdf — the preprint, 11 pages, 4 figures, 6 tables. prompt-methodology-data.zip — raw model responses (JSON Lines, one object per call), the harness that produced them, the hand-labelled gold standard of 1810 queries, a catalogue of 54 techniques with sources, and the analysis scripts. Every number, table and figure in the paper is regenerated from the raw responses by these scripts.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.