RiskyPy: Toward Automated Prompt Generation Tasks Calibrated to Model Proficiency for Joint Security and Functionality Evaluation of Generated Code

Evaluating approaches that improve the security of LLM code generation requires datasets adequate for the joint assessment of security and functionality while remaining adapted to the capabilities of the models being studied. However, existing benchmarks are costly to produce and are generally not calibrated to a chosen range of model capabilities. We introduce RiskyPy, an automated pipeline that generates security-oriented code-generation datasets calibrated to model proficiency using functionality and vulnerability canaries. We instantiate RiskyPy for Python using domains covering 18 CWEs from the 2025 CWE Top 25, producing 73 prompts. Experiments on Qwen3-4B and Seed-Coder-8B show that the generated tasks remain largely functionally accessible while frequently eliciting insecure implementations, leaving room for joint evaluation of security and functionality of security improvement approaches.