Incentive Mechanism Design in Federated Reinforcement Learning With Uniform and Non-Uniform Data Bounds

In this paper, we examine the contract design for integrating federated learning (FL) and reinforcement learning (RL). A key feature of federated reinforcement learning (FRL) is its dynamic data size, which poses challenges in incentivizing consistent user participation. Unlike traditional contracts where the contract is the same for all global training round, our design specifies the required amount of data along with the corresponding reward for each global training round. We introduce a data bound reflecting the data accumulation rate and study two scenarios: uniform data bound (same for all users) and non-uniform data bound (vary by user). For uniform data bounds, we show that monotonicity holds, simplifying individual rationality (IR) and incentive compatibility (IC) constraints. However, for non-uniform data bounds, monotonicity is violated. To address this, we propose an algorithm for non-uniform data bounds, reducing the complexity of contract design. To design the contract, we maximize the server's utility, defined as the accuracy minus the total payment, subject to individual rationality (IR) and incentive compatibility (IC), while considering data bounds. Through extensive simulations, we have validated IR and IC constraints for all types of users in every global training round. We observe that 1) the server increases data requirements as data accumulates for each type, 2) payments grow with user effort, 3) our dynamic contract model significantly outperforms existing static contract, and 4) the proposed algorithm effectively handles the complexities of non-uniform data bounds, making it a robust solution for dynamic contract design.