Machine learning prediction of rheumatoid arthritis using the gut microbiome shows limited cross-cohort generalizability

Abstract Machine-learning models trained on gut microbiome data often achieve encouraging performance within individual cohorts, but their ability to generalize across independent populations remains uncertain. Although rheumatoid arthritis (RA) is associated with alterations in gut microbial composition, it remains unclear whether these differences yield robust and transferable predictive signatures. We analyzed 16 S rRNA gut microbiome profiles from 2,238 participants, including 1,034 with RA and 1,204 healthy controls (HC). Alpha diversity, Bray–Curtis beta diversity, and differential abundance were evaluated. Four machine-learning algorithms—least absolute shrinkage and selection operator (LASSO) logistic regression, random forest, support vector machine with a radial basis function kernel, and XGBoost—were benchmarked using 638 prevalence-filtered ASVs, an 80/20 stratified train-test split, and five-fold cross-validation. Model stability was assessed across 30 repeated stratified train-test splits. Secondary genus-level analyses used an independently tuned XGBoost model to evaluate abundance-transformation sensitivity and permutation-importance stability. Leakage-free ANCOM-BC2 feature selection was performed within each training partition, with independent tuning of the reduced-feature models. For external validation, raw paired-end FASTQ data from an independent Shanghai cohort were processed using DADA2 and SILVA v138.1 and harmonized with the primary cohort at the genus-feature level; 39 samples with retained reads were evaluated. RA was associated with modest but significant reductions in Shannon diversity and observed ASV richness. Community composition differed significantly between groups (PERMANOVA P = 0.001, R² = 0.0077), although group overlap was extensive and multivariate dispersion was significantly greater in RA. Genus-level ANCOM-BC2 identified 38 robustly differentially abundant genera. Among the ASV-level models, XGBoost showed the highest observed internal discrimination, with a held-out ROC-AUC of 0.807 and a mean ROC-AUC of 0.834 across 30 repeated train-test splits. In secondary genus-level analyses, mean ROC-AUCs were 0.783 for log-transformed counts, 0.783 for relative abundance, and 0.781 for centered log-ratio representations, with no statistically significant paired differences after Holm correction. Eighteen genera appeared among the top 20 permutation-importance predictors in at least 50% of repetitions. Models using the full 447-genus feature set showed higher discrimination than independently tuned models restricted to training-only ANCOM-BC2-selected genera (mean ROC-AUC 0.782 vs. 0.740; paired Wilcoxon P = 1.83 × 10⁻⁶). After harmonization, 114 of 447 genus-level predictors (25.5%) were represented in the external cohort, including 15 of 18 stable internal predictors (83.3%). External performance remained poor (accuracy 0.410, 95% CI 0.256–0.579; sensitivity 0.450, 95% CI 0.231–0.685; specificity 0.368, 95% CI 0.163–0.616; ROC-AUC 0.439, 95% CI 0.249–0.630). Gut microbiome profiles demonstrated reproducible within-cohort predictive information for RA classification, but genus-level predictive relationships learned in the primary cohort did not transport successfully to the single independent external cohort. The lower performance of models restricted to differentially abundant genera indicates that differential-abundance significance and predictive contribution are not interchangeable. Given the small external cohort and differences in population, laboratory, sequencing, and bioinformatic procedures, the external findings should not be interpreted as evidence against generalizability of microbiome-based RA prediction more broadly. Larger multicenter studies with standardized workflows and integration of microbiome, clinical, and molecular information are needed to assess clinical transferability.