6 Exploring data scraping on ClinicalTrials.gov to identify key variables to include in an EHR-based recruitment tool

Objectives/Goals: Failure to achieve recruitment goals results in termination of ~20% of clinical trials and delays >85% of trial timelines. We aim to develop an electronic heath record (EHR)-based recruitment tool to ease identification of participants. We sought to determine whether criteria listed on clinicaltrials.gov could support selection of tool variables. Methods/Study Population: To inform the variables to include in the EHR-based recruitment tool, we data scraped clinicaltrials.gov to identify key inclusion and exclusion criteria common across a variety of diabetes clinical trials. We included actively recruiting or recently active phase 2 and 3 clinical trials of adults aged >18 years of age in the USA. We classified identified variables as clinically relevant or not and compared clinically relevant terms with inclusion and exclusion criteria (~20 variables) that were individually identified by three diabetes clinical trialists and two clinical research coordinators (CRCs). Results/Anticipated Results: We reviewed 203 clinical trials listed on clinicaltrials.gov. We identified 115 terms, 91 of which were clinically relevant. Three of 3 clinical trialists, 1 of 2 CRCs, and all trials listed age as a key variable. Consistent with data scraping, all trialists and CRCs identified glucose-lowering medications and kidney function as important criteria. Gender, ethnicity, and race were less commonly noted on clinicaltrials.gov and listed by 2 of 3 trialists and one CRC. Cardiovascular conditions (e.g., history of myocardial infarction), thyroid function tests, and contraceptive requirements were common criteria on clinicaltrials.gov, but only one trialist and one CRC identified these variables. Active infections (e.g., HIV) and c-peptide were not highlighted by trialists or CRCs but common on clinicaltrials.gov. Discussion/Significance of Impact: An EHR-based recruitment tool may facilitate identification of trial participants, but identifying key variables to include is essential. We found that data scraping for variables on clinicaltrials.gov mostly aligned with expert opinion, suggesting that automating variable selection via extraction from clinicaltrials.gov may be acceptable.

Authors

Institutions

Publication Details

Journal
Journal of Clinical and Translational Science
Published
2025-03-25
DOI
https://doi.org/10.1017/cts.2024.698
Primary Topic
Biomedical Text Mining and Ontologies
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

6 Exploring data scraping on ClinicalTrials.gov to identify key variables to include in an EHR-based recruitment tool

Emily Pfaff, Sydney Lash
Journal of Clinical and Translational Science
Biomedical Text Mining and Ontologies
article

6 Exploring data scraping on ClinicalTrials.gov to identify key variables to include in an EHR-based recruitment tool

Emily Pfaff, Sydney Lash
article en

Abstract

Objectives/Goals: Failure to achieve recruitment goals results in termination of ~20% of clinical trials and delays >85% of trial timelines. We aim to develop an electronic heath record (EHR)-based recruitment tool to ease identification of participants. We sought to determine whether criteria listed on clinicaltrials.gov could support selection of tool variables. Methods/Study Population: To inform the variables to include in the EHR-based recruitment tool, we data scraped clinicaltrials.gov to identify key inclusion and exclusion criteria common across a variety of diabetes clinical trials. We included actively recruiting or recently active phase 2 and 3 clinical trials of adults aged >18 years of age in the USA. We classified identified variables as clinically relevant or not and compared clinically relevant terms with inclusion and exclusion criteria (~20 variables) that were individually identified by three diabetes clinical trialists and two clinical research coordinators (CRCs). Results/Anticipated Results: We reviewed 203 clinical trials listed on clinicaltrials.gov. We identified 115 terms, 91 of which were clinically relevant. Three of 3 clinical trialists, 1 of 2 CRCs, and all trials listed age as a key variable. Consistent with data scraping, all trialists and CRCs identified glucose-lowering medications and kidney function as important criteria. Gender, ethnicity, and race were less commonly noted on clinicaltrials.gov and listed by 2 of 3 trialists and one CRC. Cardiovascular conditions (e.g., history of myocardial infarction), thyroid function tests, and contraceptive requirements were common criteria on clinicaltrials.gov, but only one trialist and one CRC identified these variables. Active infections (e.g., HIV) and c-peptide were not highlighted by trialists or CRCs but common on clinicaltrials.gov. Discussion/Significance of Impact: An EHR-based recruitment tool may facilitate identification of trial participants, but identifying key variables to include is essential. We found that data scraping for variables on clinicaltrials.gov mostly aligned with expert opinion, suggesting that automating variable selection via extraction from clinicaltrials.gov may be acceptable.

Journal of Clinical and Translational ScienceVol. 9(s1)
University of North Carolina at Chapel Hill (US)
Openalex Percentile: Top 95%
Biomedical Text Mining and Ontologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.