Congestion-Robust Multi-Split Computing System in Data Center Networks

In multi-split computing, to achieve low inference latency and avoid significant accuracy degradation, it is advantageous to distribute sub-models to computing nodes that can exchange intermediate data through paths with a low packet drop rate (PDR). In this paper, we first formulate an optimization problem for multi-split computing to minimize inference latency while guaranteeing sufficiently high accuracy and achieving load balancing among computing nodes. To obtain the optimal solution (i.e., optimal splitting policy) of the formulated problem, we propose a congestion-robust multi-split computing system (CRMC). In CRMC, there are two phases: 1) preparation; and 2) execution. In the preparation phase, a computing node that requires the inference result of a deep neural network model (i.e., task owner) creates accuracy prediction models according to the splitting points and the PDRs. In the execution phase, the task owner collects the amount of available computing resources of the computing nodes, the expected PDRs of paths, and the transmission rate of the wired link. Then, using the prediction models, the task owner generates possible splitting policies (i.e., candidates of the optimal splitting policy) that guarantee sufficiently high accuracy and achieve load balancing among computing nodes. Subsequently, the task owner selects the optimal splitting policy that has the lowest inference latency and executes multi-split computing by following the optimal splitting policy. Evaluation results demonstrate that CRMC effectively reduces inference latency while maintaining accuracy above 80% and ensuring balanced load distribution, with each computing node operating under 110 FLOPs (70% of the total model complexity).