Data Movement Model for the Vera C. Rubin Observatory
The sky images captured nightly by the camera on the Vera C. Rubin Observatory’s telescope will be processed across facilities on three continents. Data acquisition will occur at the observatory’s location on Cerro Pachón in the Andes mountains of Chile. A first copy of the raw image data set is stored at the summit and immediately transmitted via dedicated network links to the archive center within the US Data Facility at SLAC National Accelerator Laboratory in California, USA and from there to two European facilities for processing and archiving purposes. Data products resulting from periodic processing campaigns of the entire set of images collected since the beginning of the survey are made available to the scientific community in the form of data releases. In this paper we present an overall view of how we leverage the tools selected for managing the movement of data among the Rubin processing and serving facilities, including Rucio and FTS. We also present the tools we developed to integrate Rucio’s data model and Rubin’s Data Butler, the software abstraction layer that mediates all access to storage by pipeline tasks that implement science algorithms.
Authors
- Andrew Hanushevsky (ORCID: https://orcid.org/0009-0008-2165-0017)
- Kian-Tat Lim (ORCID: https://orcid.org/0000-0002-6338-6516)
- Mark G. Beckett (ORCID: https://orcid.org/0000-0003-3623-9753)
- Tim Jenness (ORCID: https://orcid.org/0000-0001-5982-167X)
- Stephen Pietrowicz (ORCID: https://orcid.org/0000-0002-2158-6480)
- Wei Yang (ORCID: https://orcid.org/0000-0003-4415-4020)
- Fabio Hernandez (ORCID: https://orcid.org/0000-0001-7203-2552)
- Timothy Noble
- P. A. Love (ORCID: https://orcid.org/0000-0002-7803-6674)
Institutions
- Rutherford Appleton Laboratory (GB)
- Centre National de la Recherche Scientifique (FR)
- University of Illinois Urbana-Champaign (US)
- Science and Technology Facilities Council (GB)
- SLAC National Accelerator Laboratory (US)
- Royal Observatory (GB)
- Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules (FR)
- Lancaster University (GB)
- University of Edinburgh (GB)
Publication Details
- Journal
- EPJ Web of Conferences
- Published
- 2025-01-01
- DOI
- https://doi.org/10.1051/epjconf/202533701129
- Primary Topic
- Gamma-ray bursts and supernovae
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Science Foundation
- U.S. Department of Energy
- Centre National de la Recherche Scientifique
- Science and Technology Facilities Council
- Institut National de Physique Nucléaire et de Physique des Particules
- SLAC National Accelerator Laboratory