BTVD-BERT: A Bilingual Domain-Adaptation Pre-Trained Model for Textural Vulnerability Descriptions

Textural Vulnerability Descriptions(TVD) refers to the natural language description of a vulnerability in databases like National Vulnerability Database(NVD) and China National Vulnerability Database(CNVD), which typically provides a concise summary of critical vulnerability details. To facilitate the understanding of domain-specific terms in TVD and the accurate extraction of information, we have introduced a bilingual domain-adaptation pre-trained model called BTVD-BERT, aimed at enhancing the model's capability to process and understand vulnerability descriptions in both Chinese and English. We explore three issues, the first being the impact of catastrophic forgetting on the model. Second, how should the dataset be proportioned to best enhance the model's generalization capabilities across both Chinese and English. Third, the addition of extra task-related metric information to the original dataset to construct a higher quality dataset, and whether training with this high-quality dataset can further improve model performance. We attempted to study the issues from a data engineering perspective and conducted numerous ablation experiments to find answers to these three questions. The experimental results indicate that catastrophic forgetting adversely affects the model, causing it to forget previously acquired knowledge while better retaining more recently obtained information. By employing a training approach using a mix of Chinese and English data, we were able to mitigate the impact of catastrophic forgetting on the model to some extent. Optimizing data proportions and improving data quality can effectively enhance the overall performance of the model. This study not only enhances the identification and analysis of security vulnerabilities but also offers new perspectives and empirical support for the research of multilingual domain-adaptive models.