Clustered Federated Learning with Non-IID Data: Mitigating Accuracy Overestimates Through Hold-Out Model Selection and Evaluation

Federated learning (FL), a distributed learning strategy, improves security and privacy by eliminating the need for clients to share their local data; however, FL struggles with non-IID (non independent and identically distributed) data. Clustered FL aims to remedy this by grouping similar clients and training a model per group; nevertheless, it faces difficulties in determining clusters without sharing local data and conducting model evaluation. Clustered FL evaluation on unseen clients typically applies all models, selecting the best-performer for each client - approach known as best-fit cluster evaluation. This paper challenges such evaluation process arguing that it violates a fundamental machine learning principle: test dataset labels should be used only for performance calculation, not for model selection. We show that best-fit cluster evaluation results in significant accuracy overestimates. Moreover, we present an evaluation approach that maintains the separation between model selection and evaluation by reserving a portion of the target client data for model selection, while the remaining data is used for accuracy estimation. Experiments on four datasets, encompassing various IID and non-IID scenarios, demonstrate that the best-fit cluster evaluation produces overestimates that are statistically different from our evaluation.