Decentralized Heterogeneous Federated Text Classification via Multi-Teacher Knowledge Distillation

This study tackles text classification in federated settings with highly non-IID data. We present a heterogeneous federated text classification framework driven by multi-teacher knowledge distillation. For every edge client, we label sample categories as major or minor according to their local frequency. Client models thus hold richer knowledge for their own major classes. To share this knowledge without exposing raw data, we devise a decentralized multi-teacher distillation protocol: clients exchange only distilled gradient signals, while an optimized learning algorithm aligns knowledge transfer with privacy constraints. This lightweight communication design further reduces bandwidth overhead and scales gracefully with the number of clients. Experiments on four benchmarks –Reuters-8, 20 Newsgroups, and DBPedia levels 2 and 3 –under Dirichlet-simulated heterogeneity show that our method consistently surpasses existing federated baselines in accuracy and robustness, significantly mitigating the performance drop caused by data heterogeneity.