Leveraging Resource-Aware Application-Level Checkpointing and RDMA for Fault Tolerance and Data Distribution in Malleable MPI Applications

Dynamic resource management and application malleability present numerous opportunities in High-Performance Computing (HPC), enhancing both system-level services and application performance. Recent trends in malleability research, encompassing both application and system dynamism, are building a new era in HPC. Dynamic applications, particularly those leveraging malleable resources, require adaptive checkpointing systems to enhance performance and resource use. Applications can significantly benefit from checkpointing systems becoming dynamic, especially in handling data redistribution during resource changes. Consequently, checkpointing services should also become malleable (or adaptive). Therefore, we propose iCheck, an adaptive application-level checkpoint management system that caters to malleable MPI applications. iCheck aids these applications by dynamically reconfiguring checkpointing resources and offering robust checkpointing and data redistribution services. By leveraging Remote Direct Memory Access (RDMA) to support malleable applications, iCheck facilitates faster data transfers (up to 40 times improvement over the PFS-based solution), ensuring efficient performance. The system can dynamically adjust checkpointing processes based on metrics such as available memory, checkpoint frequency, and number of processes, maintaining or improving checkpoint performance amid resource changes. This adaptive approach supports fault tolerance as well as simplifies the development of malleable applications by effectively managing resource redistribution during resource changes.