LLM2FedLLM - A Tool for Simulating Federated LLMs for Software Engineering Tasks

The paper introduces LLM2FedLLM, a tool designed for Software Engineering (SE) researchers to simulate fine-tuning Large Language Models (LLMs) within a federated learning (FL) framework. Unlike existing FL frameworks that facilitate real client collaboration, our simulator provides a controlled environment for experimenting with FL scenarios on a single machine. The LLM2FedLLM Simulator addresses SE code tasks, such as code summarization, code review, and code translation, within a federated learning framework by first partitioning the selected code dataset into heterogeneous subsets for multiple clients. It then fine-tunes the chosen LLM and evaluates its performance against vanilla, centralized, and individual client models using various metrics. The tool supports several federated aggregation methods and PEFT for supervised learning, with the flexibility to easily integrate additional techniques. The evaluation of our tool on Python code summarization showed that FedLLM performs comparably to centralized models and outperforms individual clients, particularly in low-data scenarios. Our tool aims to facilitate research advances in secure collaborative training simulations within the SE community. https://youtu.be/-byKkaiBchw.