An Overview of Ensuring Consistent Scheduler Policies Across a Multi-System Facility

An integral part of research software engineering is the system that a piece of scientific software ultimately runs. However, the system requirements for each application can differ wildly. As a result, among the vast number of facilities at a researcher’s disposal, each one can have a multitude of different systems, each catering to a unique set of requirements. An operational problem can arise in a facility when it must manage multiple systems; how can project or facility wide policies be maintained across a wide variety of systems while minimizing inconsistencies due to the mechanisms of different schedulers or architectures? At the Oak Ridge Leadership Computing Facility (OLCF), this is a consistently pertinent question. The OLCF currently has over 20 actively managed systems, and as many research projects at the OLCF use multiple systems to achieve their goals, often using different workload managers, it is important that consistency is kept between them regardless of which one is in use. While manually managing policies among multiple systems can be an option, it is far from ideal at the scale of the OLCF. There are many reasons for this, such as ensuring consistencies in time sensitive environments such as job management, or encountering difficulties in mapping policy intentions between schedulers with differing semantics. RATS CRM has been introduced in the OLCF to significantly ease this issue by managing scheduler-agnostic configurations, many of which are created automatically based on system or facility wide requirements, which are then synchronized with each system via regularly occurring jobs. These configurations ensure that time sensitive changes are handled automatically, such as applying penalties to jobs exceeding their allocation on a resource. They also greatly reduce the operational cost required to maintain consistent policies at such a scale.