High Accuracy, Low Understanding: Automated CVSS Scoring from Text

CVSS base metrics are assigned from short, free-text CVE descriptions thatare often ambiguous, yet transformer-based models are increasingly used toautomate this process. This thesis evaluates how reliably such models canpredict CVSS v3.1 metrics and whether their decisions are grounded inCVSS-relevant evidence rather than dataset artefacts.Under controlled and reproducible training conditions, encoder-only trans-formers achieve strong predictive performance, but results depend criticallyon correct model initialisation and dataset consistency. DistilBERT andBERT provide the best balance between accuracy, stability, and trainingcost, while heavier architectures are impractical for large-scale or repeateduse. Across models, Balanced Accuracy reveals systematic weaknesses onminority classes that are hidden by headline accuracy scores.To examine model behaviour, multiple interpretability methods are applied,including Integrated Gradients, SHAP, LIME, and Layer-wise RelevancePropagation. These analyses show that correct predictions frequently relyon indirect textual cues rather than explicit CVSS justification. A comple-mentary user study reveals similar uncertainty in human reasoning, indicat-ing that disagreement often stems from underspecified descriptions ratherthan model error alone. Together, the findings argue that predictive per-formance alone is insufficient to assess the reliability of automated CVSSscoring systems and support the use of transformer models as decision-support tools that expose ambiguity rather than replace expert judgement.