Improving big data analytics ecosystems using ad-hoc parallel file systems

Data processing in different technology areas, such as Artificial Intelligence or Big Data, has recently challenged High-Performance Computing (HPC). This has led to innovation in processing and managing these huge volumes of data. Numerous systems have sought to address this big-data issue. Parallel file systems, such as Expand, use techniques such as file partitioning or replication to provide highly available, high-performance storage systems for HPC environments. In addition, several frameworks have been deployed for use with parallel file systems in Big Data Analytics (BDA) environments to reduce bottlenecks caused by massive input/output (I/O) operations. This article presents a new solution for such BDA ecosystems using Apache Spark and Expand. Expand is a parallel and distributed file system designed by the ARCOS research group that can be used as a parallel ad-hoc file system to alleviate I/O bottlenecks arising in traditional parallel file systems. Using Apache Spark in conjunction with the Expand file system enables you to leverage the benefits of both platforms. Evaluations presented in this paper compare Expand with other file systems (Lustre and HDFS) and demonstrate the advantages of an ad-hoc parallel file system for BDA applications. By using the TeraSort benchmark in the evaluation, Expand was found to deliver a performance that is up to 4.5 times higher than Lustre and 2.5 times higher than HDFS. This benchmark is widely used for evaluating BDA applications.