Energy Efficiency on Cloud-Based Distributed Big Data Processing: Insights from Remote Sensing

The exponential growth in data volume and variety across various domains has led to significant challenges in efficiently processing, storing, and managing large-scale datasets. Remote sensing, with its increasingly diverse and complex datasets and demanding computational requirements, exemplifies these challenges as a prominent example of big data processing needs. Cloud computing offers a promising solution for addressing the scalability and resource allocation needs of big data processing by providing a distributed environment where resources can be dynamically managed. A typical cloud-based big data processing platform encompasses infrastructure orchestration, distributed processing frameworks, data access mechanisms, and user interfaces. While this approach enables efficient handling of large-scale data, it also raises concerns regarding energy consumption and carbon footprints. This presentation will delve into the proposed methods and tools aimed at optimizing energy consumption for big data processing within a cloud environment, using remote sensing big data as a representative example. The discussion will be organized around three interrelated topics: establishing an energy-aware benchmarking framework, optimizing infrastructure orchestration for energy efficiency, and implementing energy-efficient task scheduling. Firstly, the benchmarking framework includes applications, data, and monitoring toolkits for collecting and analyzing performance, resource utilization, and more importantly energy metrics within distributed big data systems. By benchmarking these metrics, we can identify key areas for improvement in energy efficiency. Secondly, optimizing infrastructure orchestration involves proposing resource allocation strategies such as automatic scaling of clusters, container consolidation, and prioritizing workloads considering energy efficiently as the main criterion. These strategies aim to reduce energy consumption without compromising performance as much as possible, allowing for the benefits to be applied across various big data applications without requiring changes to the existing codebase. Thirdly, a multi-objective task scheduling strategy is introduced to minimize energy consumption while maintaining acceptable execution times at computing task level. The output of this research includes software components specifically designed to be integrated to widely used remote sensing big data platforms to measure and improve the energy efficiency of distributed big data processing. Additionally, the research will engage the broader community through workshops and mini symposia to disseminate the findings and methodologies developed. By focusing on these strategies, we aim to advance the field of big data processing by providing tools that can be adapted across various domains and promote sustainable practices in cloud-based big data applications.