On Energy-aware and Verifiable Benchmarking of Big Data Processing targeting AI Pipelines

As Artificial Intelligence (AI) is revolutionizing various industries and applications, understanding the hardware requirements and energy consumption of AI pipelines in Big Data (BD) applications has become increasingly essential. This paper presents a comprehensive, scalable framework, designed to systematically measure hardware resources, energy usage, and model performance across two prominent data modalities: tabular data and images. The framework is generalizable, facilitating replicability across the AI research community, and encourages the deployment of AI models with comprehensive metrics beyond traditional accuracy, promoting the optimization of pipelines for real-world scenarios. Through detailed benchmarking, we identify EfficientNet as a standout model for image classification, and XGBoost for tabular data, both excelling in their respective domains. Notably, our findings show that Graphics Processing Units (GPUs) account for approximately 90% of total energy consumption in image-based tasks, while Central Processing Units (CPUs) are responsible for around 50% of energy use in tabular data processing. The merit of our innovative proposed framework combines information theory and probability theory to enhance our understanding of AI model performance in Edge-to-Cloud (E2C) applications that demand efficient Big Data processing in distributed environments. By seamlessly integrating energy efficiency with hardware optimization, it enables realtime monitoring of energy consumption and computing resources in containerized environments, providing precise insights for optimizing AI workloads. This framework facilitates scalable AI deployment on resource-constrained edge devices, reducing energy consumption while enhancing AI model robustness and interpretability, thereby promoting greater trust and transparency in AI-powered decision-making for critical real-world applications. This emphasizes the importance of multi-objective optimization for more sustainable and efficient Big Data AI workflows.