mdx II: A High-Performance Cloud Infrastructure for Data-Intensive and Cross-Disciplinary Science

Abstract mdx II is an Infrastructure-as-a-Service (IaaS) cloud developed to accelerate data science research and promote interdisciplinary collaboration among Japanese universities and research organizations. In contrast to conventional high-performance computing (HPC) systems, mdx II utilizes OpenStack to deliver flexible, isolated computing resources, including virtual machines, virtual networks, and storage systems. This article provides a thorough performance assessment of mdx II, benchmarking it against public cloud and bare metal server. We analyzed the performance of a 16-vCPU virtual machine in several aspects: floating-point computation, memory bandwidth, network throughput, file system and object storage access performance, and real-world application performance. When compared to an AWS 16-vCPU instance, mdx II was found to excel in many aspects, highlighting its strong potential for high-performance data analytics (HPDA) workloads. Additionally, we investigated the effects of virtualization overhead using a 224-vCPU virtual machine that fully occupies a physical host. The results revealed that virtualization overhead is negligible for compute-bound tasks, but more pronounced for memory-intensive workloads. We also evaluated the performance of a virtual machine equipped with GPUs and demonstrated that the impact of virtualization on GPU performance is negligible. These insights are anticipated to help mdx II users in achieving optimal performance for data science tasks and to guide the development of future data-centric cloud infrastructures.