Preparing the TAU performance system for exascale and beyond

The TAU Performance System ® is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, UPC, Java, Python. TAU (Tuning and Analysis Utilities) is capable of gathering performance information through instrumentation of functions, methods, basic blocks, and statements as well as event-based sampling. All C++ language features are supported including templates and namespaces. The API also provides selection of profiling groups for organizing and controlling instrumentation. The instrumentation can be inserted in the source code using an automatic instrumentation tool based on the Program Database Toolkit (PDT), dynamically using binary modification, at runtime in the Java Virtual Machine, or manually using the instrumentation API. Under the Exascale Computing Program (ECP), the TAU project was funded to prepare the software for exascale systems and beyond. Many new features and optimizations were added to TAU, including support for the new exascale system architectures and their preferred programming models. The new features include OpenMP Tools support, updated or newly implemented CUDA, HIP, and SYCL support, updated OpenACC and Clacc support, MPI updates, a new plugin API and several plugins, instrumentation updates, support for the Kokkos and Raja profiling interfaces, updated support for Python, PyTorch, TensorFlow, and Horovod, and removed threading limitations. In this paper, we will discuss these updates and more, and demonstrate the features with ECP Proxy Applications and full ECP applications.