Evaluating prompting strategies for LLM-based code generation in offensive security tasks

Context: The rapid evolution of Generative AI is transforming software engineering, enabling Large Language Models (LLMs) to generate code from natural-language prompts. This capability is increasingly applied to security-critical programming tasks, such as penetration testing, forensic analysis, traffic analysis, and OSINT. However, it remains unclear which prompt engineering strategies security analysts should adopt to obtain reliable and correct code in these scenarios. Objectives: This paper aims to systematically evaluate the impact of different prompting strategies on the quality of LLM-generated code for security-oriented tasks, assessing both functional correctness and robustness. Methods: We present a systematic evaluation of 12 prompting strategies across 4 state-of-the-art LLMs. Our evaluation framework combines textual similarity metrics with execution-based analysis conducted in a Docker-based environment, measuring correctness against ground truth on valid inputs and robustness on edge and invalid inputs. Results: Reasoning-oriented strategies, such as Chain-of-Thought and Few-Shots Prompting, consistently improve the generated code, achieving up to ＋24.6% improvement in ROUGE-L, average correctness rates of 63%–64%, and robustness rates of 50%. In contrast, refinement-based strategies such as Iterative Prompting and Question Refinement degrade structural similarity by up to -50% in BLEU-4, reducing correctness and robustness to 33% and 26%, respectively. Models producing more functionally correct code also exhibit broader behavioral coverage, reaching up to 78.24% average statement coverage. Furthermore, textual similarity alone proves insufficient to predict functional behavior. Conclusion: These findings highlight that prompt design substantially influences the reliability of LLM-generated code in security-critical scenarios, and that execution-based evaluation is essential to complement textual metrics. Reasoning-oriented prompting strategies are recommended for security analysts working with LLM-based code generation tools.