FATS: Fault-Aware Optimization for Large-Scale Model Training

As deep learning models advance into the trillion-parameter regime, thousand-GPU clusters have become commonplace for large-scale training. However, exponential system scaling introduces critical reliability challenges, where conventional fault tolerance mechanisms prove inadequate in maintaining training continuity and parameter consistency. We propose FATS (Fault-Aware Training System), an innovative framework featuring an intelligent "Monitoring-Diagnosis-Recovery" closed-loop architecture. By integrating dynamic failure prediction, distributed gradient compensation, and elastic resource management, the system demonstrates remarkable robustness improvements. Experimental results confirm the solution’s effectiveness in minimizing unplanned interruptions while preserving training efficiency under extreme-scale scenarios.