A Lightweight Approach for Adaptation of Language Models for Cybersecurity Named Entity Recognition

Cybersecurity Named Entity Recognition (CyberNER) commonly relies on fine-tuning general-purpose language models (LMs), whose WordPiece tokenizers fragment compound technical identifiers such as function names and version strings. Training from scratch with a domain-specific tokenizer addresses this at high compute cost, continual pretraining usually retains the original tokenizer, and both must be repeated as domain vocabulary evolves. We propose a lower-cost alternative: fusing a small set of domain terms into a general-purpose LM’s vocabulary before fine-tuning. Candidate terms are grouped by WordPiece split count and ranked by TF-IDF within each group, and the vocabulary budget is located across twenty configurations. On the auto-labeled STUCCO-CNC corpus (13.8 K sentences, 0.81 M tokens), fusing 250 terms that split into five or more pieces into BERT-large-cased raises 5-fold cross-validated F1 from 96.66 ± 0.25 to 96.91 ± 0.15 (best TF-IDF configuration). On the 80/20 split used to select this configuration, F1 rises from 97.20 to 97.63, exceeding four of five domain-pretrained encoders, including CySecBERT (97.45), and coming within 0.20 F1 of SecureBERT (97.83). Unlike SecureBERT, whose continual pretraining used about 1.07 billion tokens, the approach needs no pretraining run and fine-tunes in about 1.5 h (measured) on a single 24 GB GPU. Representation diagnostics (t-SNE, cluster-validity indices, cluster-wise silhouette and attention entropy) show fused-term clusters with no negatively scored members. On the manually annotated CyberNER-STIX corpus (∼23.5 K sentences, four datasets harmonized to STIX 2.1), evaluated only after the configuration was fixed on STUCCO-CNC, fusion improves entity-level F1 by about 1.0 point (1.57 points at token level), and the best configuration (80.10) exceeds the only published result on this corpus (72.30, BERT-base-cased), largely owing to the larger backbone. Fusion therefore improves on the baseline on both corpora, but the selection rule outperforms random selection only on CyberNER-STIX, so its benefit is dataset-dependent. All gains from fusion are small and indicative rather than statistically established.