Refining keyphrase extraction in academic articles via Bi-LSTM and multi-head attention

Keyphrase extraction from academic articles is an important step toward better information retrieval, indexing, and summarization in large scholarly databases. Traditional approaches, such as TF-IDF and TextRank, achieve limited success in capturing complex linguistic patterns, while more recent neural models, such as Bi-LSTM-CRF, still struggle with long-range dependencies. In this work, we propose a Bi-LSTM model with Multi-head Attention, enhanced by a custom word embedding trained on a large ArXiv corpus, and evaluate it exclusively on the abstracts of scientific articles, since abstracts condense a paper's core information and reduce noise. The model was trained on KP20 K and evaluated, at token and phrase level, across five additional corpora (ArXiv, Krapivin, Inspec, PubMed, SemEval2010), obtaining a token-level macro F1-score between 28% and 44% depending on the corpus, with no clear in-domain advantage for KP20 K over some out-of-domain corpora. We complement this evaluation with a focused ablation isolating the contribution of Multi-head Attention, preprocessing, embedding source, and class weighting, and with an empirical comparison against three fine-tuned Transformer encoders (SciBERT, BioBERT, DeBERTa). Domain-adapted Transformers substantially outperform our lighter architecture on both token- and phrase-level metrics, while DeBERTa's fine-tuning collapsed to majority-class prediction under our protocol. The contribution of this study thus lies in cross-corpus evaluation and generalization analysis rather than in state-of-the-art performance. Taken together, these experiments clarify, more than they resolve, the trade-offs between lightweight recurrent architectures and modern Transformer encoders for scientific keyphrase extraction, and the domain-generalization challenges that remain open in this task.

Authors

Institutions

Publication Details

Journal
Journal of Intelligent & Fuzzy Systems
Published
2026-11-18
DOI
https://doi.org/10.1177/18758967261509211
Primary Topic
Advanced Text Analysis Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Refining keyphrase extraction in academic articles via Bi-LSTM and multi-head attention

Jilali Antari, Abdellah El Omari
Journal of Intelligent & Fuzzy Systems
Advanced Text Analysis Techniques
article

Refining keyphrase extraction in academic articles via Bi-LSTM and multi-head attention

Jilali Antari, Abdellah El Omari
article en

Abstract

Keyphrase extraction from academic articles is an important step toward better information retrieval, indexing, and summarization in large scholarly databases. Traditional approaches, such as TF-IDF and TextRank, achieve limited success in capturing complex linguistic patterns, while more recent neural models, such as Bi-LSTM-CRF, still struggle with long-range dependencies. In this work, we propose a Bi-LSTM model with Multi-head Attention, enhanced by a custom word embedding trained on a large ArXiv corpus, and evaluate it exclusively on the abstracts of scientific articles, since abstracts condense a paper's core information and reduce noise. The model was trained on KP20 K and evaluated, at token and phrase level, across five additional corpora (ArXiv, Krapivin, Inspec, PubMed, SemEval2010), obtaining a token-level macro F1-score between 28% and 44% depending on the corpus, with no clear in-domain advantage for KP20 K over some out-of-domain corpora. We complement this evaluation with a focused ablation isolating the contribution of Multi-head Attention, preprocessing, embedding source, and class weighting, and with an empirical comparison against three fine-tuned Transformer encoders (SciBERT, BioBERT, DeBERTa). Domain-adapted Transformers substantially outperform our lighter architecture on both token- and phrase-level metrics, while DeBERTa's fine-tuning collapsed to majority-class prediction under our protocol. The contribution of this study thus lies in cross-corpus evaluation and generalization analysis rather than in state-of-the-art performance. Taken together, these experiments clarify, more than they resolve, the trade-offs between lightweight recurrent architectures and modern Transformer encoders for scientific keyphrase extraction, and the domain-generalization challenges that remain open in this task.

Journal of Intelligent & Fuzzy Systems
Université Ibn Zohr (MA)
Advanced Text Analysis Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon