A Deep Look into the Temporal I/O Behavior of HPC Applications
The increasing gap between compute and I/O speeds in high-performance computing (HPC) systems imposes the need for techniques to improve applications' I/O performance. Such techniques must rely on assumptions about I/O behavior in order to efficiently allocate I/O resources such as burst buffers, to schedule accesses to the shared parallel file system or to delay certain applications at the batch scheduler level to prevent contention, for instance. In this paper, we verify these common assumptions about I/O behavior, specifically about temporal behavior, using over 440,000 traces from real HPC systems. By combining traces from diverse systems, we characterize the behaviors observed in real HPC workloads. Among other findings, we show that I/O activity tends to last for a few seconds, and that periodic jobs are the minority, but responsible for a large portion of the I/O time. Furthermore, we make projections for the expected improvement yielded by popular approaches for I/O performance improvement. Our work provides valuable insights to everyone working to alleviate the I/O bottleneck in HPC.
Authors
- Julien Monniot (ORCID: https://orcid.org/0000-0002-2377-2666)
- Ahmad Tarraf (ORCID: https://orcid.org/0000-0002-9174-5598)
- Luan Teylo (ORCID: https://orcid.org/0000-0001-6775-2790)
- Carla Osthoff (ORCID: https://orcid.org/0000-0002-4694-7182)
- Francieli Zanon Boito (ORCID: https://orcid.org/0000-0002-1139-0724)
- André Ramos Carneiro (ORCID: https://orcid.org/0000-0002-2809-055X)
- François Tessier (ORCID: https://orcid.org/0000-0003-4441-7898)
- Mihail Popov (ORCID: https://orcid.org/0000-0002-3498-8147)
- Jakob Luettgau
- Theo Jolivel
Institutions
- Centre National de la Recherche Scientifique (FR)
- Institut national de recherche en sciences et technologies du numérique (FR)
- Université de Bordeaux (FR)
- Institut de Recherche en Informatique et Systèmes Aléatoires (FR)
- Technische Universität Darmstadt (DE)
- Laboratoire Bordelais de Recherche en Informatique (FR)
- Laboratório Nacional de Computação Científica (BR)
- Institut Polytechnique de Bordeaux (FR)
- Université de Rennes (FR)
Publication Details
- Published
- 2025-06-03
- DOI
- https://doi.org/10.1109/ipdps64566.2025.00072
- Citations
- 3
- Primary Topic
- Advanced Data Storage Technologies
- Type
- article
- Field-Weighted Citation Impact
- 4.46
Funders
- U.S. Department of Energy
- Office of Science
- Argonne National Laboratory