Developing foundations for biomedical knowledgebases from literature using large language models – A systematic assessment

While large language models (LLMs) have shown promising capabilities in biomedical applications, measuring their reliability in knowledge extraction remains a challenge. We developed a benchmark to compare LLMs in 11 literature knowledge extraction tasks that are foundational to automatic knowledgebase development, with or without task-specific examples supplied. We found large variation across the LLMs' performance, depending on the level of technical specialization, difficulty of tasks, scattering of original information, and format and terminology standardization requirements. We also found that asking the LLMs to provide the source text behind their answers is useful for overcoming some key challenges, but that specifying this requirement in the prompt is difficult.

Authors

Institutions

Publication Details

Journal
Computational and Structural Biotechnology Journal
Published
2025-01-01
DOI
https://doi.org/10.1016/j.csbj.2025.07.042
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Developing foundations for biomedical knowledgebases from literature using large language models – A systematic assessment

Sanju Sinha, Haoran Wu, Sin-Hang Fung, Qin Cao et al.
Computational and Structural Biotechnology Journal
Topic Modeling
article

Developing foundations for biomedical knowledgebases from literature using large language models – A systematic assessment

Sanju Sinha, Haoran Wu, Sin-Hang Fung, Qin Cao, Kevin Y. Yip, Stephen Kwok‐Wing Tsui, Daniel Rebibo, Jiamin Chen, Zhenghao Zhang, Miao Chen, Alfred Sze‐Lok Cheng
article en

Abstract

While large language models (LLMs) have shown promising capabilities in biomedical applications, measuring their reliability in knowledge extraction remains a challenge. We developed a benchmark to compare LLMs in 11 literature knowledge extraction tasks that are foundational to automatic knowledgebase development, with or without task-specific examples supplied. We found large variation across the LLMs' performance, depending on the level of technical specialization, difficulty of tasks, scattering of original information, and format and terminology standardization requirements. We also found that asking the LLMs to provide the source text behind their answers is useful for overcoming some key challenges, but that specifying this requirement in the prompt is difficult.

Computational and Structural Biotechnology JournalVol. 27
Chinese University of Hong Kong (HK), Sanford Burnham Prebys Medical Discovery Institute (US), Discovery Institute (US), Chinese University of Hong Kong, Shenzhen (CN)
National Natural Science Foundation of China, Chinese University of Hong Kong, National Institutes of Health, National Institute on Aging, National Cancer Institute, National Institute of General Medical Sciences
Openalex Percentile: Top 91%
Topic Modeling
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.