An AHP Framework for Evaluating Large Language Models in Software Vulnerability Assessment

Large language models (LLMs) can assist software security review, but detecting more weaknesses may increase false alarms, unreliable patches, or higher costs. This methodological paper specifies a reproducible Analytic Hierarchy Process (AHP) framework for evaluating LLM-assisted vulnerability assessment in a synthetic banking-information application. Applying the framework to a local operational banking system was not feasible because obtaining the required security approvals was difficult; no production banking infrastructure was tested. Six objectives are linked to measurable indicators covering ground-truth completeness, detection, classification and severity assessment, secure remediation, reliability and resource use, and decision sensitivity. The protocol specifies 12 vulnerable and 12 corrected variants, four scenarios, repeated runs, blinded adjudication, CVSS v4.0 assessment, regression tests, and paired analysis. A comparison with SecurityEval, CyberSecEval, CyberSecEval 2, and BountyBench positions the proposed contribution as linking verified outcomes to stakeholder priorities rather than demonstrating superior benchmark performance. An exact reciprocal AHP example uses weights of 0.40, 0.30, 0.20, and 0.10. A separate simulation yields group weights of 0.3565, 0.2920, 0.2187, and 0.1328, with a consistency ratio of 0.0008. These are constructed examples, not elicited expert judgments. No LLM benchmarking was performed, and no model ranking is claimed. Empirical validation requires archived runs, independently assessed outcomes, and human stakeholder judgments; production transfer additionally requires institutional authorization and service-specific validation.