Moko: Marrying Python with Big Data Systems

Python stands as the preferred language for data science, thanks to its user-friendly syntax and a robust ecosystem that effortlessly accommodates a variety of data types and workloads, such as relational/tabular data, tensors, and graphs. While Python thrives in smaller data settings, it struggles to scale in distributed big data environments. MOKO is an IR-based execution framework designed to extend Python's reach into the distributed big data domain by generating code that can utilize existing systems such as Spark, Dask, Torch, and GRAPE. Moko preserves Python's key features---interoperability, ease of use, and support for multi-model data types and workloads---while enabling efficient execution in a distributed setting. Our evaluation indicates that MOKO can accelerate Python applications by up to 11× across diverse systems, diminish data alignment overhead by 28×, and outperform hand-optimized solutions by 2.5×.