Evaluating large language models in biomedical data science challenges through a classroom experiment

Large language models (LLMs) have shown remarkable capabilities in algorithm design, but their effectiveness in solving data science challenges in real-world settings remains poorly understood. We conducted a classroom experiment in which graduate students used LLMs to solve biomedical data science challenges on Kaggle, focusing on tabular data prediction. While their submissions did not top the leaderboards, their prediction scores were often close to those of leading human participants. LLMs frequently recommended gradient boosting methods, which were associated with better performance. Among prompting strategies, self-refinement, where the LLM improves its own initial solution, was the most effective, a result validated using additional LLMs. While LLMs are capable of handling more complex data science tasks beyond tabular data prediction, their performance is substantially worse. These findings demonstrate that LLMs have the potential to design competitive machine learning solutions, even when used by nonexperts.

Authors

Institutions

Publication Details

Journal
Proceedings of the National Academy of Sciences
Published
2025-12-11
DOI
https://doi.org/10.1073/pnas.2521062122
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Evaluating large language models in biomedical data science challenges through a classroom experiment

Ruobing Xue, Fengnan Li, Y. Zhang, An Yuan et al.
Proceedings of the National Academy of Sciences
Artificial Intelligence in Healthcare and Education
article

Evaluating large language models in biomedical data science challenges through a classroom experiment

Ruobing Xue, Fengnan Li, Y. Zhang, An Yuan, P J Li, Leyi Sun, Hanyu Wang, Nan Cen, Zhicheng Ma, Haiqi Zhang, Tong Cheng, Tianyi Yang, Yueqi Gu, Coco Liu, Ouchan Chen, Yongyin Chen, Bing Ji, Zhicheng Ji, Vera Zhang, Anchi Yu, Yuxin Wang, Bin Liu, Xiaohui Jiang, Yutian Chen, Vivian Wang, Mengyao Shi, E.C. Ma, Jiang Shu, Rushi Tang, Austin Allen, Kym Wilson, Y. T. Liang, Vicky Shao, Tara Al-Hashimy
article en

Abstract

Large language models (LLMs) have shown remarkable capabilities in algorithm design, but their effectiveness in solving data science challenges in real-world settings remains poorly understood. We conducted a classroom experiment in which graduate students used LLMs to solve biomedical data science challenges on Kaggle, focusing on tabular data prediction. While their submissions did not top the leaderboards, their prediction scores were often close to those of leading human participants. LLMs frequently recommended gradient boosting methods, which were associated with better performance. Among prompting strategies, self-refinement, where the LLM improves its own initial solution, was the most effective, a result validated using additional LLMs. While LLMs are capable of handling more complex data science tasks beyond tabular data prediction, their performance is substantially worse. These findings demonstrate that LLMs have the potential to design competitive machine learning solutions, even when used by nonexperts.

Proceedings of the National Academy of SciencesVol. 122(50)
Duke University (US)
Openalex Percentile: Top 66%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.