Assessing Machine Learning Model Stability and the Value of Hyperparameter Transfer: Effects of Sample Size and Dataset Heterogeneity

This study investigates two critical factors affecting the performance of traditional machine learning (ML) models: training dataset size and the impact of integrating external, potentially noisy data (dataset heterogeneity). Additionally, we evaluate the effectiveness of hyperparameter transfer. Experiments utilized three distinct COVID-19 blood test datasets: the Wuhan dataset (356 cases), used as the source for hyperparameter extraction; the San Raffaele Hospital (OSR) dataset (1,624 cases), serving as the primary target for training, testing, and creating balanced subsets of varying sizes; and the Istituto Ortopedico Galeazzi (IOG) dataset (58 cases), which was integrated to simulate heterogeneity due to its collection via different devices and clinical settings. The discussion extends to a simple TL approach in which optimal hyperparameters were transferred from the Wuhan source to OSR target models. Both Gradient Boosting Machine (GBM) and Generalized Linear Model (GLM) algorithms were tested in traditional and TL settings. Performance was evaluated using accuracy, area under the curve, sensitivity, and specificity. Results show that data quantity and quality substantially influence model performance, with saturation typically observed around 450–500 observations. The addition of external, noisy data (IOG) significantly impacted specificity, particularly in GBM models trained on smaller OSR samples. Specifically, the GBM-based hyperparameter transfer learning (HTL) model demonstrated a “crossover” effect; while it initially lagged behind traditional ML in accuracy at low data volumes (0.74 vs. 0.78 at n = 50), it surpassed ML in larger datasets, reaching a peak accuracy of 0.87 (vs. 0.84 for ML) at n = 1,100. However, due to heterogeneity, this approach also exhibited sensitivity fluctuations in high-data regimes. GLM models demonstrated greater robustness with consistent accuracy levels (~0.84) and minimal variance, possibly due to their simpler structure and better alignment with underlying data characteristics. This study highlights the importance of considering both training data volume and integrity in ML and TL applications. While increasing the quantity of data can enhance performance, the unfiltered inclusion of noisy external data may degrade it. Hyperparameter transfer is a promising strategy for maintaining model reliability across varying data conditions.

Authors

Institutions

Publication Details

Journal
Acta Infologica
Published
2026-11-13
DOI
https://doi.org/10.26650/acin.1726175
Primary Topic
Machine Learning and Data Classification
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Assessing Machine Learning Model Stability and the Value of Hyperparameter Transfer: Effects of Sample Size and Dataset Heterogeneity

Volkan KAPUCU, Eralp Doğu
2 links
Acta Infologica
Machine Learning and Data Classification
article

Assessing Machine Learning Model Stability and the Value of Hyperparameter Transfer: Effects of Sample Size and Dataset Heterogeneity

Volkan KAPUCU, Eralp Doğu
article en

Abstract

This study investigates two critical factors affecting the performance of traditional machine learning (ML) models: training dataset size and the impact of integrating external, potentially noisy data (dataset heterogeneity). Additionally, we evaluate the effectiveness of hyperparameter transfer. Experiments utilized three distinct COVID-19 blood test datasets: the Wuhan dataset (356 cases), used as the source for hyperparameter extraction; the San Raffaele Hospital (OSR) dataset (1,624 cases), serving as the primary target for training, testing, and creating balanced subsets of varying sizes; and the Istituto Ortopedico Galeazzi (IOG) dataset (58 cases), which was integrated to simulate heterogeneity due to its collection via different devices and clinical settings. The discussion extends to a simple TL approach in which optimal hyperparameters were transferred from the Wuhan source to OSR target models. Both Gradient Boosting Machine (GBM) and Generalized Linear Model (GLM) algorithms were tested in traditional and TL settings. Performance was evaluated using accuracy, area under the curve, sensitivity, and specificity. Results show that data quantity and quality substantially influence model performance, with saturation typically observed around 450–500 observations. The addition of external, noisy data (IOG) significantly impacted specificity, particularly in GBM models trained on smaller OSR samples. Specifically, the GBM-based hyperparameter transfer learning (HTL) model demonstrated a “crossover” effect; while it initially lagged behind traditional ML in accuracy at low data volumes (0.74 vs. 0.78 at n = 50), it surpassed ML in larger datasets, reaching a peak accuracy of 0.87 (vs. 0.84 for ML) at n = 1,100. However, due to heterogeneity, this approach also exhibited sensitivity fluctuations in high-data regimes. GLM models demonstrated greater robustness with consistent accuracy levels (~0.84) and minimal variance, possibly due to their simpler structure and better alignment with underlying data characteristics. This study highlights the importance of considering both training data volume and integrity in ML and TL applications. While increasing the quantity of data can enhance performance, the unfiltered inclusion of noisy external data may degrade it. Hyperparameter transfer is a promising strategy for maintaining model reliability across varying data conditions.

Acta InfologicaVol. 0(0)
Muğla University (TR)
Good health and well-being
Machine Learning and Data Classification
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon