Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE
The requirement classification technique helps with planning, quality assurance, architecture decisions, and risk management. However, traditional metrics regard all misclassifications as equally costly. In this work, we evaluate Term Frequency-Inverse Document Frequency (TF-IDF) with Logistic Regression, TF-IDF with LinearSVC, Qwen3.5-4B, and Gemma 4 E2B IT on PROMISE_exp in two tasks: binary functional/non-functional classification and 11-class NFR subtype classification. In addition to common metrics, we provide four risk metrics: Risk-Weighted Error (RWE), Criticality Miss Rate (CMR) by triage level, Critical-Alert Precision (CAP), and Critical Subtype Error Rate (CSER). Uncertainty is measured with 2,000 paired class-stratified bootstrap replicates. LinearSVC achieves the highest accuracy both in Task A (0.907) and Task B (0.752), and the lowest Task B RWE (0.235). On gold-labelled NFRs, Qwen3.5-4B shows the best criticality-triage point estimates: CMR=0.073, CAP=0.911, and CSER=0.127. Its paired CMR and CSER differences against Gemma exclude zero, whereas its comparisons with the two classical baselines remain uncertain. Sensitivity analysis reveals that this advantage of Qwen3.5-4B does not persist when the six critical classes are considered equally important. Its differences against Gemma lose support when Gemma's five invalid outputs are removed. When Task A errors are included in triage, Qwen misses 19/55 critical requirements compared with 7/55 for LinearSVC. Thus, risk metrics give the most meaningful results in combination with class-specific, output validity, and pipeline-level analysis.
Authors
- Abdullah Önden (ORCID: https://orcid.org/0000-0003-3769-8193)
- Temel Kaan Ekiz (ORCID: https://orcid.org/0009-0007-1699-4301)
Institutions
- Istanbul University (TR)
Publication Details
- Journal
- Applied Research Advances
- Published
- 2026-11-16
- DOI
- https://doi.org/10.65069/ara31202741
- Primary Topic
- Software Engineering Research
- Type
- article
- Field-Weighted Citation Impact
- 0.00