Self-Optimizing Distributed Cloud Computing with Dynamic Neural Resource Allocation and Fault-Tolerant Multi-Agent Systems

The exponential growth of distributed cloud systems necessitates an intelligent form of workload management that ensures optimum performance, scalability, and reliability. Traditional approaches based on static or heuristic resource allocation and fault-tolerance mechanisms cannot capture the dynamic nature of modern workloads and heterogeneous infrastructures. Drawing from this source of inspiration, this work develops a holistic framework integrating machine learning models that facilitate self-optimizing distributed cloud computing for dynamic workload management. Three components are significant in the proposed framework: (1) DNRA; this uses a hybrid GNN and RL architecture for real-time resource allocation, offering up to 25% improvement in resource utilization and a 30% reduction in task latency. (2) Workload-Aware Transfer Learning Scheduler (WATLS), leveraging transfer learning to predict workload dynamics across heterogeneous nodes, achieving 20% higher task throughput and 15% energy savings through proactive scheduling. (3) Multi-Agent Predictive Fault-Tolerant System (MAP-FTS), combining multi-agent systems with predictive fault detection to preemptively redistribute tasks and schedule maintenance, reducing system downtime by 50% and enhancing fault detection accuracy to 90%. The framework integrates novel methodologies towards answering the main challenges of resource allocation, workload prediction, and fault tolerance. The amount of progress achieved through cumulative contributions includes improvement in system reliability and scalability up to 5x their nodes and respective scalable energy efficiency without any degradation of performance. This work paves the pathway for an adaptive and intelligent next generation of distributed cloud systems.