Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory
The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.
Authors
- Greg Daues
- Jennifer Adelman-McCarthy
- B. R. White (ORCID: https://orcid.org/0000-0001-8619-3234)
- Kian-Tat Lim (ORCID: https://orcid.org/0000-0002-6338-6516)
- W. Guan (ORCID: https://orcid.org/0000-0002-5548-5194)
- Tim Jenness (ORCID: https://orcid.org/0000-0001-5982-167X)
- Stephen Pietrowicz (ORCID: https://orcid.org/0000-0002-2158-6480)
- Wei Yang (ORCID: https://orcid.org/0000-0003-4415-4020)
- Zhaoyu Yang (ORCID: https://orcid.org/0009-0001-8401-5087)
- Fabio Hernandez (ORCID: https://orcid.org/0000-0001-7203-2552)
- E. Karavakis (ORCID: https://orcid.org/0000-0002-5729-5167)
- Yuyi Guo
- B. Yanny
- Michelle Gower
- Timothy Noble
- R. Dubois
- P. A. Love (ORCID: https://orcid.org/0000-0002-7803-6674)
Institutions
- Rutherford Appleton Laboratory (GB)
- Centre National de la Recherche Scientifique (FR)
- Fermi National Accelerator Laboratory (US)
- University of Illinois Urbana-Champaign (US)
- Science and Technology Facilities Council (GB)
- Brookhaven National Laboratory (US)
- SLAC National Accelerator Laboratory (US)
- Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules (FR)
- Lancaster University (GB)
Publication Details
- Journal
- EPJ Web of Conferences
- Published
- 2025-01-01
- DOI
- https://doi.org/10.1051/epjconf/202533701234
- Primary Topic
- Astronomy and Astrophysical Research
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Science Foundation
- U.S. Department of Energy
- Centre National de la Recherche Scientifique
- Science and Technology Facilities Council
- Institut National de Physique Nucléaire et de Physique des Particules
- SLAC National Accelerator Laboratory