In Silico World - D6.3 Efficiency recovery strategies report

The purpose of this deliverable is to report on strategies adopted in the ISW project to ensure operational efficiency and smooth processing of large-scale workflows with the use of research models developed in the project, and through the Model Execution Environment (MEE) (1), identifying and resolving performance bottlenecks which arise when computations are submitted to the available High Performance Computing resources. The cornerstone of the developed solution is rich monitoring of the scientific applications and the HPC computing infrastructure based on the modern observability principles. Observability adopts a holistic approach, using a suite of methodologies, strategies, and tools designed to gather and analyze detailed telemetry data. The primary goal of observability is to uncover unrecognized issues and investigate the ``unknown unknowns'' that plague the underlying computer systems and applications (2).Existing observability technologies emerged in the context of cloud-native systems and commercial applications, hence their adoption to scientific applications in HPC environments is not straightforward. Key challenges include collection of application-level metrics, instrumentation of scientific applications to collect rich telemetry data, and tracing of distributed call chains in the HPC execution ecosystem. The solution described in this document is based on open-source standard and tools: the OpenTelemetry (3) observability framework, the OpenSearch (4) data search and analysis engine, Grafana (5), and JupyterHub (6) providing a notebook-based environment for advanced analysis of the collected telemetry data. The remainder of this document is structured as follows. Section 2 briefly describes the scientific applications monitored using the proposed system. Section 3 introduces the architecture and implementation details of the monitoring system. Section 4 presents examples of performance monitoring and analytical capabilities of developed system. Section 5 highlights optimization conducted based on the data analytics capabilities of the monitoring system. Section 6 concludes work.