Benchmarking cell type and gene set annotation by large language models with AnnDictionary

We develop an open-source package called AnnDictionary to facilitate the parallel, independent analysis of multiple anndata. AnnDictionary is built on top of LangChain and AnnData and supports all common large language model (LLM) providers. AnnDictionary only requires 1 line of code to configure or switch the LLM backend and it contains numerous multithreading optimizations to support the analysis of many anndata and large anndata. We use AnnDictionary to perform the first benchmarking study of all major LLMs at de novo cell-type annotation. LLMs vary greatly in absolute agreement with manual annotation based on model size. Inter-LLM agreement also varies with model size. We find that LLM annotation of most major cell types to be more than 80-90% accurate, and will maintain a leaderboard of LLM cell type annotation. Furthermore, we benchmark these LLMs at functional annotation of gene sets, and find that Claude 3.5 Sonnet recovers close matches of functional gene set annotations in over 80% of test sets.

Authors

Institutions

Publication Details

Journal
Nature Communications
Published
2025-10-28
DOI
https://doi.org/10.1038/s41467-025-64511-x
Citations
4
Primary Topic
Machine Learning and Algorithms
Type
article
Field-Weighted Citation Impact
9.92
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Benchmarking cell type and gene set annotation by large language models with AnnDictionary

Thomas A. Rando, Ke Ding, Aris Taychameekiatchai, Baoxiang Li et al.
4 citations
Nature Communications
Machine Learning and Algorithms
9.92
article

Benchmarking cell type and gene set annotation by large language models with AnnDictionary

Thomas A. Rando, Ke Ding, Aris Taychameekiatchai, Baoxiang Li, Thomas H. Slone, Madhav Mantri, James Zou, William Harper, Elvira Mennillo, Elena Montauti, Hanbing Song, Sean P. Spencer, Alkis J. Psaltis, Wenfei Sun, Robrecht Cannoodt, Angela M. Detweiler, Simon Bucher, Patrick Neuhöfer, Annie Jensen, Antoine de Morrée, Sheela Crasta, J. Quintanilla, Xiaochen Fan, Rebecca N. Culver, Michelle Griffin, Robert C. Jones, Bruce Wang, Elisa B. Caffrey, Maurizio Morri, Gyehyun Lee, Keliana Hui, Patricia K. Nguyen, William R. Goodyer, Kim Chi Vo, I Rusu, Justin Hong, Frank M. McCarthy, Linda C. Giudice, Varun Ramanan Subramaniam, A.D. Choudhury, Emily Trimm, Francisco X. Galdos, Nir Yosef, Lauren Duan, Albert Y. Wu, Douglas E. Henze, Michael Swift, Alexander J. Tarashansky, Luca Ghita, Sheryl Paul, Julian Berg, Kyle Awayan, Matthew Choi, Haibing Hua, Sevahn K. Vorperian, Steven E. Artandi, Mark A. Krasnow, Lynn Yang, Philip A. Beachy, Hyun Kang, Jun He, Karen Gabriel Gonzales, David Seong, Wan-Jin Lu, Shravani Mukherjee, Andres Gottfried‐Blackmore, Bronwyn Scott, Bhavani A. Sanagavarapu, Serena Y. Tan, Ivana Cvijović, R. Ponnusamy, James M. Gardner, Stewart J. Kim, Sivakamasundari Vijayakumar, Samuel D’Souza, Abhishek Murti, Leah C. Dorman, Christin S. Kuo, Zue Zhang, Angela H. Ling, Yuchao Gu, George Crowley, Karim Mrouj, Aaron McGeever, Zev J. Gartner, Soochi Kim, Taha Adnan Jan, Lindsay S. Moore, Rebeca Arroyo Hornero, Stefan Veizades, Dhruv Bhatt, Norma Neff, David Yoonsuk Oh, Hala Shakib Dhowre, SoRi Jang, Mariko H. Foecke, Deana R.C. Colburg, Pierre Boyeau, Sara Billings, Rebecca Chen, Mallory A. Laboulaye, Juan C. Irwin, Saswati Karmakar, Eliza A. Gaylord, Honey Mekonen, Aditi Swarup, Michael G. Kattah, Aditi Agrawal, Sean M. Wu, Kiet T. Phong, Astrid Gillich, Marcus L. Forst, Juliana Idoyaga, Hayley M. Raquer-McKay, Angela Oliveira Pisco, Kavita Murthy, Michael F. Clarke, Ahmad Salehi, Gregory L. Szot, Jennifer Parker, Diana J. Laird, Julia Salzman, Franklin W. Huang, Balint Forgo, Ryan M. Samuel, Can Ergen, C. S. Tang, Paul V. Heinrich, Alan Gi-Lun Cheng, Alina A. Isakova, Honor Paine, Luise A. Seeker, Santiago Guerrero, Ting-Hsuan Wu, Stephen R. Quake, Juliane Winkler, Pratima Prabala, Yinghua Xie, Daniel Lee, Tian Wang, Dong Hua, Zhen Qi, Bo Yu, Chen Dong, Anping Li, Jia Yan, Shuo Han, Jengmin Kang, Julia H. Pham, Siyu He, Zifa Wang, Sunny Nguyen, Amanda Seng, R.C. Garner, Sandy Mahfouz, Jaeyoon Lee, Burnett S. Kelly, J. L. Corrales Aguirre, Amber Casillas, William Kong, Kristy Red-Horse, Ivan Sampson, Ho‐Su Sin, William Yue, Michael T. Longaker, Nikita Sajai, Bikem Soyur, Resham Irfan, Marcela Alcántara-Hernádez, Jamie Bozeman, Kenneth Donaville, Elise Lelou, Ali Reza Rais Sadati
article en
4 citations

Abstract

We develop an open-source package called AnnDictionary to facilitate the parallel, independent analysis of multiple anndata. AnnDictionary is built on top of LangChain and AnnData and supports all common large language model (LLM) providers. AnnDictionary only requires 1 line of code to configure or switch the LLM backend and it contains numerous multithreading optimizations to support the analysis of many anndata and large anndata. We use AnnDictionary to perform the first benchmarking study of all major LLMs at de novo cell-type annotation. LLMs vary greatly in absolute agreement with manual annotation based on model size. Inter-LLM agreement also varies with model size. We find that LLM annotation of most major cell types to be more than 80-90% accurate, and will maintain a leaderboard of LLM cell type annotation. Furthermore, we benchmark these LLMs at functional annotation of gene sets, and find that Claude 3.5 Sonnet recovers close matches of functional gene set annotations in over 80% of test sets.

Nature CommunicationsVol. 16(1)
San Francisco VA Medical Center (US), Howard Hughes Medical Institute (US), Palo Alto University (US), University of California, San Francisco (US), Aarhus University (DK), VA Palo Alto Health Care System (US), Cardiovascular Institute of the South (US), California Department of Health Care Services (US), Ragon Institute of MGH, MIT and Harvard (US), Ghent University (BE), Reproductive Science Center (US), Stanford Health Care (US), Donor Network West (US), Chan Zuckerberg Initiative (United States) (US), Stanford Medicine (US), VIB-UGent Center for Inflammation Research (BE), Fluidda (Belgium) (BE), San Francisco VA Health Care System (US), Broad Center (US), San Diego Unified School District (US), Medical University of Vienna (AT), Columbia University (US), Vanderbilt University Medical Center (US), University of California, Berkeley (US), Stanford University (US)
Openalex Percentile: Top 2%
Machine Learning and Algorithms
9.92
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.