Developing a cloud based near real time system for large scale water quantity and quality models

The eReefs project is a multi-agency collaboration aimed at providing situational and retrospective information on the Great Barrier Reef (GBR). The Bureau of Meteorology have developed water quantity and quality models for the 459,000 km GBR catchment area. A distributed hydrological model known as Grid-to-Grid was used for water quantity modelling, generating hourly 1 km grid outputs spanning 459,000 grids. For water quality modelling, multivariate regression statistical models have been developed. The outputs of these river models are used as input for hydrodynamic and biogeochemical ocean models. These provide simulations of the marine environment and contribute to the preparation of the Reef Report Card which captures the health of the Reef. The Bureau developed two systems using these models, one operated manually to generate long historical simulations using the National Computational Infrastructure, and another operating daily to generate near real time simulations of water quantity and quality. Developing and deploying these models presented many challenges including data handling, ensuring data consistency and availability, managing costs, and providing operational resilience. We have adopted Amazon Web Services to address these challenges, implementing a system using Managed Workflow for Apache Airflow (MWAA). MWAA defines workflows through tasks that call the elastic container service, executing python container images containing the models processing code. These services were created using Terraform, which provide infrastructure as code. This is the first such large scale water model deployed to the cloud in the Bureau, enabling further insights on catchment based threats to the health of the GBR.