Heterogeneous Distributed Data Management in Academia

Comprehensive data management - access control, storage, and replication - is a challenge even when dealing with a single-site on premises architecture. For medium-to-large academic projects, distributed across many institutes and computing systems, this challenge is significantly heightened. With such a heterogeneous computing setup - from hardware, to networking, and policy - custom software is necessitated. At the same time, academics are loath to give up their POSIX file systems and ‘offline’ access to data. In this talk, I will describe the Simons Observatory’s data management framework that provides access for our 1 PB/year data rates to 500 users across three continents and four core computing nodes (national labs and university clusters). I will specifically note the significant architectural decisions that were necessary given our academic infrastructure: from containerisation to the use of trusted networks like Globus to power data transfers, and the need for significant caching to empower traditional academic workflows. The Advanced Simons Observatory software suite is fully open source, and I will signpost to our solutions throughout.