DeNAV: Decentralized Self-Supervised Learning with a Training Navigator

Abstract Federated learning has gained significant attention for enabling effective model training on distributed data. However, in practice, most client data is unlabeled, and reliable communication with a central server is often infeasible at scale. To address this, we consider a more realistic setting where clients hold only unlabeled data and can communicate only with neighbors and propose DeNAV, a novel decentralized self-supervised learning framework for handling such scenarios. DeNAV simultaneously pre-trains multiple lightweight transformer models across clients. To improve training, we design a navigator algorithm to smartly plan the training route of each model and adopt staleness-aware model aggregation to handle the discrepancy of training status between models. DeNAV eliminates the need for server coordination, offering both convergence and consensus guarantees. Extensive experiments show that DeNAV is comparable to state-of-the-art federated self-supervised learning baselines and also surpasses previous decentralized methods with equal communication efficiency.