Advancing mixed-effects random forests for predictions based on multi-cohort data

Pooling longitudinal cohorts can extend age coverage and improve long-term health predictions, but cohort heterogeneity complicate estimation and prediction. Mixed-effects machine learning (ME-ML) combines flexible machine learning methods with mixed-effect models to account for correlated observations, yet these ME-ML methods are limited in their applicability in complex settings such as multi-cohort data. We propose a multi-cohort mixed-effects random forest (MERFmulti-cohort) for improving predictions of individual health trajectories beyond the observed period. The method extends standard MERF through (i) better separation of population-level and subject-specific variation, (ii) flexibility in correcting systematic discrepancies in fixed-effects estimates to improve covariance parameter estimation, and (iii) flexible specification of random-effects and residual variance-covariance structures. We evaluate MERFmulti-cohort against linear regression, linear mixed models, random forests, and standard MERF in simulations and four real-data scenarios predicting BMI z-scores in children and adolescents. In simulations, MERFmulti-cohort improves covariance parameter estimation across several covariance structures while achieving prediction accuracy comparable to standard MERF. Predictive gains are most obvious when residual variances vary across study waves. In real-data applications, MERFmulti-cohort improves predictive accuracy relative to MERF by flexibly capturing complex patterns and accommodating cohort heterogeneity. Pooling similar cohorts covering extended age coverage can improve future individual-level predictions if the heterogeneity in multi-cohort data is adequately accounted for.

Authors

Publication Details

Journal
arXiv (Cornell University)
Published
2026-11-18
DOI
https://doi.org/10.48550/arxiv.2611.24463
Primary Topic
Statistical Methods and Bayesian Inference
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Advancing mixed-effects random forests for predictions based on multi-cohort data

Jiumeng Zhang, Claudia Börnhorst, Iris Pigeot, Tanja Schultz et al.
arXiv (Cornell University)
Statistical Methods and Bayesian Inference
preprint

Advancing mixed-effects random forests for predictions based on multi-cohort data

Jiumeng Zhang, Claudia Börnhorst, Iris Pigeot, Tanja Schultz, Jordan Behrendt, Tanja Vrijkotte, Siyu Zhou
preprint en

Abstract

Pooling longitudinal cohorts can extend age coverage and improve long-term health predictions, but cohort heterogeneity complicate estimation and prediction. Mixed-effects machine learning (ME-ML) combines flexible machine learning methods with mixed-effect models to account for correlated observations, yet these ME-ML methods are limited in their applicability in complex settings such as multi-cohort data. We propose a multi-cohort mixed-effects random forest (MERFmulti-cohort) for improving predictions of individual health trajectories beyond the observed period. The method extends standard MERF through (i) better separation of population-level and subject-specific variation, (ii) flexibility in correcting systematic discrepancies in fixed-effects estimates to improve covariance parameter estimation, and (iii) flexible specification of random-effects and residual variance-covariance structures. We evaluate MERFmulti-cohort against linear regression, linear mixed models, random forests, and standard MERF in simulations and four real-data scenarios predicting BMI z-scores in children and adolescents. In simulations, MERFmulti-cohort improves covariance parameter estimation across several covariance structures while achieving prediction accuracy comparable to standard MERF. Predictive gains are most obvious when residual variances vary across study waves. In real-data applications, MERFmulti-cohort improves predictive accuracy relative to MERF by flexibly capturing complex patterns and accommodating cohort heterogeneity. Pooling similar cohorts covering extended age coverage can improve future individual-level predictions if the heterogeneity in multi-cohort data is adequately accounted for.

arXiv (Cornell University)
Statistical Methods and Bayesian Inference
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon