Secure Domain Adaptation and Retrieval Augmented Generation for Compact Language Models Using Archived Cybersecurity Intelligence

Abstract Security analysts must reconcile vulnerability, exploit and malware records that are scattered across public repositories, yet it is unclear whether compact, locally deployable language models can reason over such archives. We present SecureLLM, an open framework that normalizes 161,190 records from the National Vulnerability Database, the Known Exploited Vulnerabilities catalog, ExploitDB and MalwareBazaar into one corpus, which we use both to continue pretraining Qwen2.5-1.5B-Instruct and to build a dense retrieval index. We compare the adapted model with its base and three other compact models on 1,000 stratified security prompts, with and without retrieval (10,000 responses), using paired non-parametric tests. Continued pretraining lowered validation loss by 12% during training but left cybersecurity-term coverage in closed-book answers statistically unchanged (Holm-adjusted p = 0.062). With retrieval, the adapted model's answers agreed less with the retrieved context than those of its base (cosine similarity 0.41 versus 0.46; adjusted p < 0.001). On a balanced 10,000-sample EMBER malware test set, gradient-boosted trees reached 0.856 accuracy, whereas the zero-shot language models reached 0.646 (base) and 0.630 (adapted), with precision near 0.91 but recall of only 0.32 and 0.29. In a 100-sample pilot, adding retrieved-neighbor statistics to the prompt made the model label every sample benign. We trace these outcomes to four causes: a mismatch between raw-text pretraining and instruction-style use, a mismatch between the prompt set and the corpus, encoder truncation, and uninformative retrieval neighborhoods. We release the pipeline and propose a leakage-controlled evaluation protocol for cyber-intelligence language models.