Cloud-Driven AI: Enhancing Machine Learning Performance Through Distributed Computing

The exponential data volume and complexity of Machine Learning (ML) algorithms has resulted in an increase in computational limitations, which affects artificial intelligence [AI]. This research seeks to establish whether cloud-based distributed computing can effectively address these limitations and improve the performance of ML. Our approach is a new architecture that deploys the computing load over a cloud framework in order to distribute and process in parallel some of the computational majorities of learning algorithms. To further enhance the application’s efficiency of processing the circuit data, a combination of adaptive resource allocation techniques, data partitioning strategies, and a new synchronization protocol has been proposed and implemented. This framework was assessed with deep neural networks, random forests, gradient-boosting machines to different datasets, and complex tasks. Focusing on the obtained results, it becomes possible to state that the proposed methods yield up to 87% improvement in the training speed and 5–12% increase in the accuracy of the models compared to the traditional implementations in single nodes. Moreover, our approach proved its capability of scalability did not degrade the performance and achieved near-linear speedup throughout the distributed nodes of up to 1000. These results propose that cloud-based AI can significantly reduce time to on-ramp for training numerous models on larger datasets. In light of this, distributed computing in the cloud offers a way to meet the computational demands of today’s AI applications while opening the door to new advancements in ML throughout different domains.