Making Earth System Modelling and Observation Workflows Reproducible and Portable Across Federated High-Performance Computer and Cloud Platforms with Conda-based Containers

Earth System Modeling (ESM) workflows, which are critical for Earth Observation (EO) and climate research, are highly complex and demand substantial computational power. These workflows typically employ domain decomposition techniques, using a combination of MPI (Message Passing Interface) and OpenMP (Open Multi-Processing) to parallelize computations efficiently. For global high-resolution models, each processor handles a small column of data (e.g., latitude-longitude and vertical levels), enabling scalability on powerful supercomputers like the LUMI EuroHPC. Conversely, calculations for smaller domains can be performed on personal computers, underscoring the need for a portable approach that spans this computational spectrum. This work presents the use of Conda, an open-source package manager, to create reproducible, portable, and scalable computational environments for ESM workflows across federated HPC and cloud platforms. A key challenge was that default Conda MPI implementations were not optimized for the high-performance interconnects and architectures of modern supercomputers. They lacked in particular support for advanced communication libraries such as Universal Communication X (UCX) for Mellanox Infiniband-based clusters and Open Fabric Interfaces (OFI) for Intel Omni-Path and other RDMA-capable networks like the EuroHPC. To address this, we upgraded the MPI toolchains in Conda-Forge, ensuring compatibility and optimized performance on state-of-the-art HPC infrastructure. This approach significantly accelerates the development of new algorithms by allowing scientists to develop locally on their personal computers or even laptops, then seamlessly scale to the largest HPC systems without having to modify their codes and scripts due to differences in the software stack. This eliminates the typically time-intensive porting steps required to adapt workflows to HPC environments, drastically shortening the transition from research development to production. This process should not only accelerate innovation by streamlining the full software environment management and usage but also make it more accessible to non-specialists, empowering researchers to focus on scientific discovery rather than sorting out technical software compatibility issues. Conda environments seamlessly run on a range of platforms without relying on host software stacks or requiring privilege escalation. By integrating these environments into lightweight containers, it is possible to achieve performance that matches or even exceeds native configurations. Additionally, conda-based containers, viewed as single compressed files, reduce storage requirements and alleviate strain on storage and file systems. Through comprehensive case studies and benchmarks, we demonstrate how this approach supports reproducibility, scalability, and computational efficiency for ESM workflows. This work highlights the transformative potential of Conda and containerized solutions in enabling complex EO modeling, accelerating innovation, and fostering sustainable and scalable digital innovation across federated platforms.