Refining keyphrase extraction in academic articles via Bi-LSTM and multi-head attention
Keyphrase extraction from academic articles is an important step toward better information retrieval, indexing, and summarization in large scholarly databases. Traditional approaches, such as TF-IDF and TextRank, achieve limited success in capturing complex linguistic patterns, while more recent neural models, such as Bi-LSTM-CRF, still struggle with long-range dependencies. In this work, we propose a Bi-LSTM model with Multi-head Attention, enhanced by a custom word embedding trained on a large ArXiv corpus, and evaluate it exclusively on the abstracts of scientific articles, since abstracts condense a paper's core information and reduce noise. The model was trained on KP20 K and evaluated, at token and phrase level, across five additional corpora (ArXiv, Krapivin, Inspec, PubMed, SemEval2010), obtaining a token-level macro F1-score between 28% and 44% depending on the corpus, with no clear in-domain advantage for KP20 K over some out-of-domain corpora. We complement this evaluation with a focused ablation isolating the contribution of Multi-head Attention, preprocessing, embedding source, and class weighting, and with an empirical comparison against three fine-tuned Transformer encoders (SciBERT, BioBERT, DeBERTa). Domain-adapted Transformers substantially outperform our lighter architecture on both token- and phrase-level metrics, while DeBERTa's fine-tuning collapsed to majority-class prediction under our protocol. The contribution of this study thus lies in cross-corpus evaluation and generalization analysis rather than in state-of-the-art performance. Taken together, these experiments clarify, more than they resolve, the trade-offs between lightweight recurrent architectures and modern Transformer encoders for scientific keyphrase extraction, and the domain-generalization challenges that remain open in this task.
Authors
- Jilali Antari (ORCID: https://orcid.org/0000-0003-3084-9753)
- Abdellah El Omari
Institutions
- Université Ibn Zohr (MA)
Publication Details
- Journal
- Journal of Intelligent & Fuzzy Systems
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1177/18758967261509211
- Primary Topic
- Advanced Text Analysis Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00