Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE

The requirement classification technique helps with planning, quality assurance, architecture decisions, and risk management. However, traditional metrics regard all misclassifications as equally costly. In this work, we evaluate Term Frequency-Inverse Document Frequency (TF-IDF) with Logistic Regression, TF-IDF with LinearSVC, Qwen3.5-4B, and Gemma 4 E2B IT on PROMISE_exp in two tasks: binary functional/non-functional classification and 11-class NFR subtype classification. In addition to common metrics, we provide four risk metrics: Risk-Weighted Error (RWE), Criticality Miss Rate (CMR) by triage level, Critical-Alert Precision (CAP), and Critical Subtype Error Rate (CSER). Uncertainty is measured with 2,000 paired class-stratified bootstrap replicates. LinearSVC achieves the highest accuracy both in Task A (0.907) and Task B (0.752), and the lowest Task B RWE (0.235). On gold-labelled NFRs, Qwen3.5-4B shows the best criticality-triage point estimates: CMR=0.073, CAP=0.911, and CSER=0.127. Its paired CMR and CSER differences against Gemma exclude zero, whereas its comparisons with the two classical baselines remain uncertain. Sensitivity analysis reveals that this advantage of Qwen3.5-4B does not persist when the six critical classes are considered equally important. Its differences against Gemma lose support when Gemma's five invalid outputs are removed. When Task A errors are included in triage, Qwen misses 19/55 critical requirements compared with 7/55 for LinearSVC. Thus, risk metrics give the most meaningful results in combination with class-specific, output validity, and pipeline-level analysis.

Authors

Institutions

Publication Details

Journal
Applied Research Advances
Published
2026-11-16
DOI
https://doi.org/10.65069/ara31202741
Primary Topic
Software Engineering Research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE

Abdullah Önden, Temel Kaan Ekiz
3 links
Applied Research Advances
Software Engineering Research
article

Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE

Abdullah Önden, Temel Kaan Ekiz
article en

Abstract

The requirement classification technique helps with planning, quality assurance, architecture decisions, and risk management. However, traditional metrics regard all misclassifications as equally costly. In this work, we evaluate Term Frequency-Inverse Document Frequency (TF-IDF) with Logistic Regression, TF-IDF with LinearSVC, Qwen3.5-4B, and Gemma 4 E2B IT on PROMISE_exp in two tasks: binary functional/non-functional classification and 11-class NFR subtype classification. In addition to common metrics, we provide four risk metrics: Risk-Weighted Error (RWE), Criticality Miss Rate (CMR) by triage level, Critical-Alert Precision (CAP), and Critical Subtype Error Rate (CSER). Uncertainty is measured with 2,000 paired class-stratified bootstrap replicates. LinearSVC achieves the highest accuracy both in Task A (0.907) and Task B (0.752), and the lowest Task B RWE (0.235). On gold-labelled NFRs, Qwen3.5-4B shows the best criticality-triage point estimates: CMR=0.073, CAP=0.911, and CSER=0.127. Its paired CMR and CSER differences against Gemma exclude zero, whereas its comparisons with the two classical baselines remain uncertain. Sensitivity analysis reveals that this advantage of Qwen3.5-4B does not persist when the six critical classes are considered equally important. Its differences against Gemma lose support when Gemma's five invalid outputs are removed. When Task A errors are included in triage, Qwen misses 19/55 critical requirements compared with 7/55 for LinearSVC. Thus, risk metrics give the most meaningful results in combination with class-specific, output validity, and pipeline-level analysis.

Applied Research Advances
Istanbul University (TR)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon