TDML — A Trustworthy Distributed Machine Learning Framework

Recent years have witnessed a surge in deep learning research, marked by the introduction of expensive generative models like OpenAI’s SORA and GPT, Meta AI’s LLAMA series, and Google’s FLAN, BART, and Gemini models. However, the rapid advancement of large models (LM) has intensified the demand for computing resources, particularly GPUs, which are crucial for their parallel processing capabilities. This demand is exacerbated by limited GPU availability due to supply chain delays and monopolistic acquisition by major tech firms. Distributed Machine Learning (DML) methods, such as Federated Learning (FL), mitigate these challenges by partitioning data and models across multiple servers, though implementing optimizations like tensor and pipeline parallelism remains complex. Blockchain technology emerges as a promising solution, ensuring data integrity, scalability, and trust in distributed computing environments, but still lacks guidance on building practical DML systems. In this paper, we propose a trustworthy distributed machine learning (TDML) framework that leverages blockchain to coordinate remote trainers and validate workloads, achieving privacy, transparency, and efficient model training across public remote computing resources. Experimental validation demonstrates TDML’s efficacy in overcoming performance limitations and malicious node detection, positioning it as a robust solution for scalable and secure distributed machine learning. • Efficient Distributed, Trustworthy, Secure Computing Framework: (i) Proposed a new framework that utilizes blockchain technology to coordinate and verify the workloads of remote trainers in distributed ML to achieve privacy, transparency, and data traceability. • Byzantine Attack Detection and Defense: (i) To overcome the Byzantine attacks from malicious nodes, we proposed malicious detection and incentive-driven mechanisms, respectively. (ii) Conducted experiments verified that our gradient-based malicious detection effectively identifies and isolates malicious trainers from benign nodes. • Comprehensive Performance Evaluation: (i) Conducted comprehensive experiments showcasing the effectiveness of the proposed framework TDML against three baseline techniques. (ii) Experimental results show that our TDML framework surpasses the performance limitations of FedAvg and matches the baseline performance of single-node training.