Adaptive Bayesian transfer learning under compositional generalized linear mixed model for microbiome data analysis

With the development of high-throughput sequencing technologies, microbiome data play an increasingly important role in pathological analysis and disease prediction. However, microbiome data are typically characterized by high dimensionality, small sample sizes, and compositional constraints, which present challenges to conventional statistical methods, particularly in studies of specific diseases with limited sample sizes. Motivated by these challenges, this paper develops an adaptive Bayesian transfer learning framework built upon the Bayesian Compositional Generalized Linear Mixed Model (BCGLMM). The proposed framework integrates a power-prior-based adaptive Bayesian transfer learning with BCGLMM, which enables information borrowing from a related source dataset while accounting for compositional constraints, phylogenetic relatedness among taxa, and dependence among samples in high-dimensional microbiome data. By adopting an adaptive transfer parameter, the framework can adjust the contribution of source information according to the compatibility between the source and target datasets, thereby improving robustness and reducing the risk of negative transfer. Systematic simulation studies show that the proposed Bayesian transfer learning methods generally achieve better predictive performance than target-only, pooled, and existing transfer-learning competitors, especially when the target sample size is limited and related source data are available. Compared with the fixed-transfer method, Adaptive BTL-CGLMM maintains more stable performance under increasing source-target discrepancy, different dimensions, varying sample sizes, and heterogeneous covariance structures. Real gut microbiome applications, including body mass index (BMI) prediction, type 2 diabetes (T2D) diagnosis, and colorectal cancer (CRC) diagnosis, further demonstrate that the proposed methods improve prediction by borrowing useful auxiliary information from related source datasets. The microbial taxa identified by Adaptive BTL-CGLMM are also consistent with existing biological findings, supporting the interpretability of the proposed framework. This study provides an adaptive Bayesian transfer learning framework for high-dimensional microbiome data analysis with limited target sample sizes. By enabling knowledge transfer under compositional constraints, the proposed method improves predictive performance and robustness across varying data settings, thereby enhancing phenotype prediction and disease diagnosis in small-sample microbiome studies.