The Unassailable Case for PCIe and CXL in Single-Root Scale-Up Systems

This white paper presents a definitive, technically rigorous argument to unequivocally prove the following statement: If your goal is to build a scale-up accelerator system under a single root complex, Ethernet, RoCE, or PCIe-over-Ethernet are not just inefficient—they are fundamentally wrong tools. These technologies violate the physics, semantics, and timing model that make a single-root system possible. The correct and only viable technologies for this architecture are PCIe Gen5/6 and CXL 3.0 (or NVLink/NVSwitch in GPU-centric designs). All other interconnects, including Ethernet, RoCE, SmartNICs, and DPUs, belong outside the server, connecting nodes together, not replacing the bus inside them. This paper will demonstrate, with scientific precision and hardware-level evidence, that Ethernet-based solutions cannot technically or physically substitute for PCIe/CXL in a single-root system. We will provide quantitative comparisons, signal flow diagrams, and latency/power measurements from verified public specifications and white papers to make this case unassailable. The frequent misunderstanding of this topic stems from the conflation of scale-up (intra-node) and scale-out (inter-node) architectures, a confusion this paper will decisively clear.