Towards Cloud-Native Workflows for Environmental Data: Lessons from NOC and UKCEH

Scientific datasets are continuing to increase in size. Traditional methods of data analysis are struggling with the increased local storage and compute resources required. As a result, a transition to cloud-based technologies and ways of working is becoming increasingly necessary. This talk shares the ongoing collaboration between the UK’s National Oceanography Centre (NOC) and the UK Centre for Ecology and Hydrology (UKCEH) to explore management, sharing and analysis of large gridded environmental datasets based in the cloud. Our goal is to make the transition to working with datasets on the cloud as easy as possible for researchers. We will focus on three key areas: object storage: what it is and why we are using it, cloud-native data formats and how we convert between them at scale, and interfaces for visualising and working with the data. We store our data in object storage systems using Zarr and Kerchunk formats, which support massively-parallel access over the web and thus scalable computations. We’ve tested different ways to convert data, and will share what works well and what doesn’t. Using these formats come with its own set of challenges however. Key amongst these is the requirement that data is split into chunks. Chunk size strongly influences speed of data processing and visualisation but chunk sizes that are good for scientific analysis are not good for visualisation. We will share how we have attempted to mitigate this and other related challenges. We’ll show benchmarks comparing different methods and chunking strategy performance on the data conversion, simple scientific analyses and the tile-server visualisation, and share successes, challenges and tips that folk can take away from our work so far. We hope it will help others who are thinking about or working towards making their data systems more cloud-ready.A recording of this session is available on YouTube: https://youtu.be/LDg2shyY5EI