Using Globus as the high-performance data transport mechanism at Monash University

Current in-house solutions have been pivotal in Monash University's data storage and transfer infrastructure; however, the data management requirements have outgrown currently used systems, particularly in their ability to provide data reliable ingest from the instruments generating large datasets and large volumes of data, ability to easily move the data between the storage and computing facilities, to automate the data processing pipelines, and easily share and collaborate on data. We ve been investigating various data orchestration solutions and deployed Globus as a pilot solution of choice. We report on our journey in trialling Globus. We will share our experience, challenges, and will discuss the deployed solutions across the Monash network and Monash eResearch infrastructure.

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2024-10-30
DOI
https://doi.org/10.5281/zenodo.15293628
Primary Topic
Distributed and Parallel Computing Systems
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Using Globus as the high-performance data transport mechanism at Monash University

Zenodo (CERN European Organization for Nuclear Research)
Distributed and Parallel Computing Systems
article

Using Globus as the high-performance data transport mechanism at Monash University

article en

Abstract

Current in-house solutions have been pivotal in Monash University's data storage and transfer infrastructure; however, the data management requirements have outgrown currently used systems, particularly in their ability to provide data reliable ingest from the instruments generating large datasets and large volumes of data, ability to easily move the data between the storage and computing facilities, to automate the data processing pipelines, and easily share and collaborate on data. We ve been investigating various data orchestration solutions and deployed Globus as a pilot solution of choice. We report on our journey in trialling Globus. We will share our experience, challenges, and will discuss the deployed solutions across the Monash network and Monash eResearch infrastructure.

Zenodo (CERN European Organization for Nuclear Research)
Industry, innovation and infrastructure
Openalex Percentile: Top 58%
Distributed and Parallel Computing Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.