Do Rule-Derived Labels Improve Prediction When Reported Labels Are Limited? A Two-Site Machine Learning Study

**Background.** Rule derived (weak) labels can enlarge a training set when reported labels are scarce. Their value depends on label quality, on how the extra records shift the training distribution, and on whether the comparison accounts for reported labels set aside for tuning or calibration. **Methods.** We ran an exploratory label-budget experiment on *Escherichia coli* urine cultures from two de-identified electronic health record datasets: Stanford ARMD and UT Southwestern ARMD-UTSW. The task was to predict reported non-susceptibility to cefepime, cefoxitin and ceftazidime (Stanford) and to cefepime and ceftazidime (UT Southwestern). Reported-label budgets of 10%, 25%, 50% and 100% of development patients were drawn over five patient-disjoint outer splits. Logistic regression and histogram gradient boosting were trained with reported labels only, or with added rule-derived labels using equal weights, a tuning-selected weight, class-proportion adjustment or prior-probability correction. These were compared with an all-budget reported-only baseline that used every budgeted reported label for training. Stanford used the inferred labels released with the dataset and a feature set including resistance history. UT Southwestern used researcher-derived labels from Stanford rule text, which were not validated for that site, and an age-only primary model. Brier-score differences were assessed with paired patient-cluster bootstrap intervals. **Results.** All rule-derived labels were susceptible. Equal-weight augmentation lowered mean predicted risk well below observed prevalence (for example, Stanford cefepime at the 10% budget: 0.35% versus 4.85%) and had a higher Brier score than the all-budget baseline in every drug, model and budget summary. The tuning-selected weight did not consistently improve on the all-budget baseline. Its median Brier difference was positive in 38 of 40 drug, model and budget summaries. Its paired intervals lay above zero in 100 of 296 Stanford analyses and 31 of 198 UT Southwestern analyses, and below zero in 3 and 5. The tuning set often selected a weight of zero (146 of 296 and 153 of 198 analyses). At UT Southwestern, the age-only models showed little discrimination (median AUROC 0.49 to 0.54), so those comparisons mainly reflect calibration. **Conclusions.** Under these designs, rule-derived labels did not consistently improve probability prediction over a reported-only model that used the whole reported-label budget for training. Evaluations of weak supervision should include such a baseline and should report calibration as well as discrimination. **Keywords:** weak supervision; limited-label learning; label budget; probability calibration; class imbalance; antimicrobial resistance

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-13
DOI
https://doi.org/10.64898/2026.11.12.26367890
Primary Topic
Machine Learning in Healthcare
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Do Rule-Derived Labels Improve Prediction When Reported Labels Are Limited? A Two-Site Machine Learning Study

Talim Hossain Sadi, Chowdhury Aseer Ruthbah
medRxiv
Machine Learning in Healthcare
preprint

Do Rule-Derived Labels Improve Prediction When Reported Labels Are Limited? A Two-Site Machine Learning Study

Talim Hossain Sadi, Chowdhury Aseer Ruthbah
preprint en

Abstract

**Background.** Rule derived (weak) labels can enlarge a training set when reported labels are scarce. Their value depends on label quality, on how the extra records shift the training distribution, and on whether the comparison accounts for reported labels set aside for tuning or calibration. **Methods.** We ran an exploratory label-budget experiment on *Escherichia coli* urine cultures from two de-identified electronic health record datasets: Stanford ARMD and UT Southwestern ARMD-UTSW. The task was to predict reported non-susceptibility to cefepime, cefoxitin and ceftazidime (Stanford) and to cefepime and ceftazidime (UT Southwestern). Reported-label budgets of 10%, 25%, 50% and 100% of development patients were drawn over five patient-disjoint outer splits. Logistic regression and histogram gradient boosting were trained with reported labels only, or with added rule-derived labels using equal weights, a tuning-selected weight, class-proportion adjustment or prior-probability correction. These were compared with an all-budget reported-only baseline that used every budgeted reported label for training. Stanford used the inferred labels released with the dataset and a feature set including resistance history. UT Southwestern used researcher-derived labels from Stanford rule text, which were not validated for that site, and an age-only primary model. Brier-score differences were assessed with paired patient-cluster bootstrap intervals. **Results.** All rule-derived labels were susceptible. Equal-weight augmentation lowered mean predicted risk well below observed prevalence (for example, Stanford cefepime at the 10% budget: 0.35% versus 4.85%) and had a higher Brier score than the all-budget baseline in every drug, model and budget summary. The tuning-selected weight did not consistently improve on the all-budget baseline. Its median Brier difference was positive in 38 of 40 drug, model and budget summaries. Its paired intervals lay above zero in 100 of 296 Stanford analyses and 31 of 198 UT Southwestern analyses, and below zero in 3 and 5. The tuning set often selected a weight of zero (146 of 296 and 153 of 198 analyses). At UT Southwestern, the age-only models showed little discrimination (median AUROC 0.49 to 0.54), so those comparisons mainly reflect calibration. **Conclusions.** Under these designs, rule-derived labels did not consistently improve probability prediction over a reported-only model that used the whole reported-label budget for training. Evaluations of weak supervision should include such a baseline and should report calibration as well as discrimination. **Keywords:** weak supervision; limited-label learning; label budget; probability calibration; class imbalance; antimicrobial resistance

medRxiv
BRAC University (BD)
Good health and well-being
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon