AICB: A benchmark for evaluating the communication subsystem of LLM training clusters

AICB (Artificial Intelligence Communication Benchmark) is a benchmark for evaluating the communication subsystem of GPU clusters, which includes representative workloads in the fields of Large Language Model (LLM) training. Guided by the theories and methodologies of Evaluatology, we simplified the real-workload LLM training systems through AICB that maintain good representativeness and usability. AICB bridges the gap between application benchmarks and microbenchmarks in the scope of LLM training. In addition, we constructed a new GPU-free evaluation system that helps researchers evaluate the communication system of the LLM training systems. To help the urgent demand on this evaluation subject, we open-source AICB and make it available at https://github.com/aliyun/aicb. • Guided by the principles of Evaluatology, we propose AICB, a benchmark to evaluate AI communication systems. By “hijacking” the training framework, we construct a pragmatic Evaluation System and develop a GPU-Free system through simulation. • We use the elapsed time of every specific workload as metrics to evaluate the LLM communication subsystem. Through the assessment of case studies, we demonstrate the practicality of AICB in evaluating corresponding communication performance. • Beyond the communication behavior, AICB can output the LLM training communication workload, serving as input for SimAI to simulate the overall performance of the cluster training.