Optimized Hybrid Framework vs. Spark and Hadoop: Performance Analysis for Big Data Applications in Vehicular Engine Systems

The rapid growth of data generated by vehicular engine systems has necessitated the development of efficient frameworks for real-time big data processing. Existing solutions such as Apache Hadoop and Apache Spark, have been widely adopted in various domains due to their respective strengths; Hadoop’s distributed storage capabilities and Spark’s in-memory processing efficiency. However, these standalone frameworks present limitations when applied to vehicular systems, particularly in handling real-time analytics, low-latency requirements, and efficient use of network and energy resources. Previous research, including approaches like the one by Shabestari and Navimipour (2023), which combines Spark with Hadoop YARN for remote sensing data, and Ahmed et al. (2020), which investigates parameter tuning for Hadoop and Spark, highlights the difficulties in achieving optimal performance in dynamic environments, particularly when managing varying data loads, noise levels, and non-numeric attributes. This paper introduces a novel Optimized Hybrid Framework that integrates Hadoop’s distributed storage and Spark’s in-memory processing while addressing the limitations of existing hybrid models. Unlike prior approaches, our framework incorporates a dynamic switching mechanism based on business logic, data urgency, and noise levels, providing an adaptive and efficient solution for real-time vehicular big data analytics. Moreover, the framework introduced controlled noise injection strategies and handling for non-numeric attributes such as decision and driver behavior data, ensuring data integrity in real-world vehicular datasets. Experimental results demonstrate that the proposed Hybrid Framework outperforms both Hadoop and Spark in key performance metrics, achieving an execution time of 0.030 seconds, latency of 1.67e-7 seconds/row, and a network load of 55.01%. These results highlight the framework’s ability to optimize throughput, minimize latency, and reduce energy consumption, making it particularly suitable for predictive maintenance, fault diagnosis, and adaptive vehicular control systems. This study contributes to the field by offering a practical, scalable, and low-latency solution for vehicular big data analytics, addressing the shortcomings of both standalone and hybrid approaches in the automotive manufacturers domain, fleet operators, and researchers for a more effective tool for vehicle health monitoring. The enhanced data processing capabilities facilitate better vehicle maintenance, increase safety, and reduce operational costs, delivering substantial benefits to the broader community through more dependable and efficient transportation solution.