Does Linux Provide Performance Isolation for NVMe SSDs? Configuring cgroups for I/O Control in the NVMe Era

Modern storage workloads commonly run in containers within data centers, such as machine learning, databases, caches, HPC, and serverless workloads. To facilitate the storage performance requirements (e.g., bandwidth, latency) of these workloads, data centers have adopted fast NVMe SSDs as a storage medium. At the same time, data centers virtualize and share these storage resources with multiple tenants to improve resource utilization and reduce costs. Such sharing leads to an inherent trade-off between tenant performance isolation and SSD utilization. Although various research studies demonstrate how to achieve various performance isolation properties, such as fairness, there is neither a unified definition for performance isolation nor a benchmark. Furthermore, the isolation capabilities of state-of-the-practice I/O control mechanisms in the Linux kernel are not well understood. In this paper, we address these three challenges. First, we survey the definition of performance isolation and uncover four common performance isolation desiderata. Second, we introduceisol-bench, a benchmark for evaluating these desiderata for I/O control mechanisms. Third, we use isol-bench to evaluate I/O isolation for Linux’s state-of-the-practice I/O control mechanism, cgroups. From our evaluation, we are able to conclude that out of cgroups’s knobs io.cost achieves the most isolation desiderata, but has a latency overhead past CPU saturation. We open-source the source code of isol-bench at https://github.com/atlarge-research/isol-bench.