Domain-Adaptive Pre-Training of Language Models for Text Mining of Cybersecurity Documents

This study demonstrates the effectiveness of Domain-Adaptive Pre-Training (DAPT) on pre-trained language models for cyber-security text mining. Experiments using various versions of BERT and RoBERTa confirm significant improvements in domain-specific tasks, particularly in identifying security-related concepts and terminology, and reveal that DAPT's effectiveness varies depending on the label type.

Authors

Institutions

Publication Details

Journal
IEICE Transactions on Information and Systems
Published
2025-08-25
DOI
https://doi.org/10.1587/transinf.2024ofl0002
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Domain-Adaptive Pre-Training of Language Models for Text Mining of Cybersecurity Documents

Masami Mohri, Thin Tharaphe Thein, Yoshiaki Shiraishi, Keitaro Aota et al.
1 links
IEICE Transactions on Information and Systems
Topic Modeling
article

Domain-Adaptive Pre-Training of Language Models for Text Mining of Cybersecurity Documents

Masami Mohri, Thin Tharaphe Thein, Yoshiaki Shiraishi, Keitaro Aota, Makoto Takita, Hiroki Kuzuno
article en

Abstract

This study demonstrates the effectiveness of Domain-Adaptive Pre-Training (DAPT) on pre-trained language models for cyber-security text mining. Experiments using various versions of BERT and RoBERTa confirm significant improvements in domain-specific tasks, particularly in identifying security-related concepts and terminology, and reveal that DAPT's effectiveness varies depending on the label type.

IEICE Transactions on Information and SystemsVol. E108.D(12)
Kobe Design University (JP), Institute of Informatics of the Slovak Academy of Sciences (SK), Kobe University (JP)
Openalex Percentile: Top 89%
Topic Modeling
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Domain-Adaptive Pre-Training of Language Models for Text Mining of Cybersecurity Documents — Masami Mohri, Thin Tharaphe Thein, et al. · IEICE Transactions on Information and Systems (2025) | TGRS Research Map | TGRS