Optimization of distributed compute resources utilization in the CMS Global Pool

The CMS Submission Infrastructure is the primary system for managing computing resources for CMS workflows, including data processing, simulation, and analysis. It integrates geographically distributed resources from Grid, HPC, and cloud providers into federated pools managed by HTCondor and Glidein- WMS, for a total of around 500k CPU cores. This system dynamically manages workloads based on priorities defined by the collaboration. Additionally, CMS scheduling strategies must be flexible to handle multiple concurrent workloads while considering changing processing demands and resource availability from various providers. Efficient utilization of vast amounts of distributed compute resources is a key element for the success of the scientific programs of the LHC experiments. Optimizing the system is essential to maximize resource efficiency and fully utilize the distributed computing power. The CMS Submission Infrastructure team thus systematically investigates sources of inefficiency in workload scheduling to reduce their impact. In addition, a strategy of pilot overloading has been introduced to compensate for other inefficiency sources, thereby optimizing resource utilization and enhancing computational throughput.

Authors

Publication Details

Journal
Springer Link (Chiba Institute of Technology)
Published
2025-10-07
DOI
https://doi.org/10.1051/epjconf/202533701210/pdf
Primary Topic
Distributed and Parallel Computing Systems
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Optimization of distributed compute resources utilization in the CMS Global Pool

Hyunwoo J. Kim, A. Pérez-Calero Yzquierdo, Marco Mascheroni, Vaiva Zokaite et al.
Springer Link (Chiba Institute of Technology)
Distributed and Parallel Computing Systems
article

Optimization of distributed compute resources utilization in the CMS Global Pool

Hyunwoo J. Kim, A. Pérez-Calero Yzquierdo, Marco Mascheroni, Vaiva Zokaite, Frank Würthwein, Florian Von Cube
article en

Abstract

The CMS Submission Infrastructure is the primary system for managing computing resources for CMS workflows, including data processing, simulation, and analysis. It integrates geographically distributed resources from Grid, HPC, and cloud providers into federated pools managed by HTCondor and Glidein- WMS, for a total of around 500k CPU cores. This system dynamically manages workloads based on priorities defined by the collaboration. Additionally, CMS scheduling strategies must be flexible to handle multiple concurrent workloads while considering changing processing demands and resource availability from various providers. Efficient utilization of vast amounts of distributed compute resources is a key element for the success of the scientific programs of the LHC experiments. Optimizing the system is essential to maximize resource efficiency and fully utilize the distributed computing power. The CMS Submission Infrastructure team thus systematically investigates sources of inefficiency in workload scheduling to reduce their impact. In addition, a strategy of pilot overloading has been introduced to compensate for other inefficiency sources, thereby optimizing resource utilization and enhancing computational throughput.

Springer Link (Chiba Institute of Technology)
Industry, innovation and infrastructure
Openalex Percentile: Top 27%
Distributed and Parallel Computing Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Optimization of distributed compute resources utilization in the CMS Global Pool — Hyunwoo J. Kim, A. Pérez-Calero Yzquierdo, et al. · Springer Link (Chiba Institute of Technology) (2025) | TGRS Research Map | TGRS