BigOPERA: An OPportunistic and Elastic Resource Allocation for big data frameworks

Abstract Efficient asset management is essential for optimizing the performance and scalability of modern Big Data (BD) frameworks. However, traditional resource allocation methods often suffer from static partitioning, inefficient resource utilization, and high operational costs, limiting their ability to adapt to fluctuating workloads dynamically. This paper introduces BigOPERA, an opportunistic and elastic resource allocation framework designed to enhance BD processing environments by integrating dedicated and non-dedicated computing assets. Leveraging containerization and a two-tiered scheduling mechanism, BigOPERA dynamically manages available resources to improve workload execution efficiency. Experimental results demonstrate that BigOPERA achieves up to 35% performance improvement over native Apache Spark configurations, significantly enhancing computational throughput while optimizing resource consumption. Our findings highlight the potential of BigOPERA in scalable, cost-effective, and sustainable BD processing.