The Gather Join: Efficiently Aligning Atmospheric Datasets in Apache Spark

Data processing in climate research regularly involves analyzing massive datasets, particularly atmospheric model data that spans around the globe. A typical task is to align this model data with measurement series for validation or trend analysis. Unfortunately, for this task, traditional join operations prove extremely inefficient: They load vast amounts of data into main-memory only to discard most of it afterwards due to the high selectivity typically occurring in these joins.To address this issue, we introduce a new operation we call the Gather Join. This method optimizes highly selective join operations by directly loading only the portions of the larger dataset that are actually relevant to the join result. We achieve this by exploiting the spatio-temporal information that is typically encoded in the file structure of the dataset. We present three different strategies for realizing the Gather Join, where we integrate the operation in the cluster computing framework Apache Spark to handle the multidimensional atmospheric datasets stored in NetCDF files. Our results show that the Gather Join outperforms the traditional join operation by a factor of up to 135x for the tested datasets. Increasing the selectivity of the join query is expected to further enhance these performance gains.