Accelerating massive data processing in Python with Heat

Processing massive scientific datasets is challenging. In astrophysics as in the vast majority of research communities, the standard approach involves breaking up and analyzing data in smaller chunks, an inefficient and very prone-to-errors process. The problem is exacerbated on GPUs, because of the smaller available memory. Popular solutions to distribute NumPy/SciPy computations are based on task parallelism, introducing significant runtime overhead, complicating implementation, and often limiting GPU support to one vendor. In this talk, I will present an alternative based on data parallelism. The open-source library Heat [https://github.com/helmholtz-analytics/heat] builds on PyTorch and the Message Passing Interface protocol to simplify porting of NumPy/SciPy-based code to multi-CPU, multi-GPU clusters. From a user perspective, Heat can be used seamlessly in the Python array ecosystem (numpy, scipy, xarray etc.). Under the hood, massive memory-intensive operations are distributed transparently via MPI communication. I will not bore you with technical details, but rather show you practical situations in which your Python pipeline could benefit from using Heat as a backend for massive array manipulations, statistics, signal processing, machine learning, etc.