FedEnsemble: federated learning model for efficient sentiment analysis

Abstract In the realm of Natural Language Processing (NLP), sentiment analysis plays a crucial role in deriving insights from vast amounts of user-generated text. However, deploying state-of-the-art transformer-based models in real-world applications poses two key challenges: preserving user data privacy and mitigating the computational overhead associated with large-scale models. This study introduces FedEnsemble, a novel federated learning framework that addresses these challenges through three core innovations: (i) a heterogeneous ensemble of BERT, RoBERTa, and DistilBERT to enhance classification robustness; (ii) an entropy-based attention stacking mechanism that adaptively fuses model outputs according to predictive confidence; and (iii) Quantization-Aware Training (QAT) to compress models while maintaining high accuracy and communication efficiency. Extensive experiments on multiple benchmark datasets—including IMDB, Amazon Reviews, AG News, Sentiment140, and ISOT Fake News—demonstrate the effectiveness and generalizability of the proposed model. FedEnsemble consistently outperforms baseline meta-learners, achieving up to 98.10% accuracy on the IMDB dataset while delivering competitive results across diverse and noisy data distributions. Furthermore, applying 4-bit QAT reduced model size by 86.90% and inference time by 48.69%, with only a 1.80% decrease in accuracy. Experiments under non-IID conditions with skewed label distributions further confirmed the robustness of the model, maintaining accuracy above 95%. Statistical significance tests (t-test, p < 0.05) validated the reliability of these improvements.