Balancing Performance and Productivity: A Comparative Study of Apache Arrow vs. MPI

As large-scale data processing becomes increasingly essential in today’s world, balancing developer productivity and computational performance in high-performance computing (HPC) environments remains a persistent challenge. Conventional HPC workloads rely heavily on MPI-based solutions, often written in C for the better performance. Meanwhile, Apache Arrow, specifically leveraging Arrow Flight for node-to-node data transfers, has gained traction as a flexible, in-memory columnar data approach that promises efficient, language-agnostic memory usage, although its usage in HPC environments is less exploredIn this paper, we perform a preliminary study that consists of three different implementations of the same data-parallel workload — the "Monte Carlo Simulation for Financial Risk Assessment" — in (1) MPI in C, (2) MPI in Python, and (3) Python-based Arrow. Specifically, we measure and compare the trade-offs between performance and lines of code for each implementation. In addition, we demonstrate the scalability of each implementation through a strong-scaling analysis, highlighting potential communication bottlenecks as we add more communicators. The results of this paper are intended to surface early trends for balancing developer productivity, raw performance, and scalability when selecting tools for large-scale computational tasks.