GFLLM: Federated Learning in Large Language Models With Optimized Group Aggregation

Large language model (LLM) have shown their potential to improve the efficiency and effectiveness of medical works. On the other hand, developments in federated learning (FL) have made it possible to utilize medical textual datasets in different data silos to fulfilling the need to improve medical insight while addressing stringent privacy and security concerns. However, fine-tuning LLMs in FL results in high communication costs due to frequent transmissions between the FL server and the client and a large number of trainable parameters. To solve this problem, we propose GFLLM: Group Aggregation for Federated Learning in Large Language Models. Our method first assigns the clients into groups and performs partial model aggregation within the group, and then the clients with larger bandwidth with the cloud server upload the weights. The results of simulation experiments demonstrate that our proposed method saves 27.5 % of the time on average over traditional FL when facing LLMs with 7 billion parameters.