Federated LLM Pre-Training on Mobile Phones

Over the past decades, billions of mobile phones have become the primary interface to the Internet, accessing vast amounts of private user data. These devices are idle most of the time or become obsolete after a few years. Pre-training Large Language Models (LLMs) on ubiquitous mobile phones offers a promising way to utilize both private data and idle computing power. In this work, we propose the first federated LLM pre-training framework for mobile devices and demonstrate that it can potentially achieve wall-clock training time comparable to centralized pre-training.