TailorFS: An Adaptive File System to Support Dynamic I/O requirements of HPC Workloads

High-Performance Computing (HPC) systems typically provide a global storage system to support large-scale workload’s I/O requirements. However, these workloads have diversified from traditional simulation to include big data analytics and AI workloads. HPC systems include hardware accelerators and storage software with configurations to support this workload diversification. However, selecting the correct combination of these configurations for each workload is a complex task even for I/O experts. To address this problem, we designed TailorFS, a software abstraction using FSView that transparently selects the appropriate file system characteristics including software abstractions, hardware accelerators, and optimizations to accelerate I/O for a given workload. Given a workload’s behavior, TailorFS chooses an appropriate software existing in the system and a corresponding configuration to dynamically optimize the workload. With TailorFS, users can utilize any interface to perform I/O on the file system and get the correct software selected for them based on the described behavior. Finally, TailorFS can accelerate I/O for different workload types by up to 46 × by utilizing workload characteristics. In conclusion, TailorFS can accelerate complex HPC workflows by up to 6.5 × better I/O performance on the Lassen supercomputer by using dynamically created workload-aware FSViews.