Enhancing Real-Time Geospatial Big Data Processing and Presentation Through Cloud-Optimised Storage and Formats

This talk will explore a transformative approach to handling extensive geospatial data through a streamlined and scalable system architecture. Our method hinges on four foundational elements: cloud-optimised data, object stores, dynamic tile servers, and static web interfaces. We discuss transitioning from traditional scientist-favoured formats like NetCDF and GeoTIFF to advanced, cloud-optimised formats such as Cloud Optimised GeoTIFF and Zarr. These formats are compatible with popular data processing libraries like xarray and rasterio, facilitating parallel processing and efficient access without requiring complete downloads. Data storage is achieved using S3-compatible object stores, exemplified by systems like JASMIN (The UK's data analysis facility for environmental science). These stores facilitate the management of substantial datasets by utilising parallelized file systems capable of multiple concurrent reads and supporting HTTP range requests for partial data fetching. This capability is critical for integrating with cloud-optimised files efficiently. Additionally, we introduce a novel approach for on-the-fly data tiles generation for frontend applications, utilizing libraries such as rioxarray and rasterio. This enables the deployment of lightweight tile server applications on serverless architectures, dramatically reducing the system's operational footprint. This talk will also highlight real-world applications and projects that currently utilize this architecture, demonstrating its effectiveness in processing and presenting multi-gigabyte data files through static web applications. This comprehensive system not only simplifies data handling, but also improves scalability and accessibility, making it a model for future data-intensive applications. A recording of this session is available on YouTube: https://youtu.be/DdHsSzFxw8E