Analysis of Air Quality Index using Apache Spark Framework

Air Quality Index (AQI) analysis is crucial for understanding and addressing air pollution problems. This paper emphasizes the importance of AQI analysis, particularly using the Apache Spark framework. The Apache Spark framework is highlighted for its ability to manage large-scale AQI data and perform machine learning tasks using MLlib. This paper describes how to build machine learning pipelines in Apache Spark and apply them to dataset. The overall analysis shows the importance of AQI analysis within the Apache Spark framework and performs the precision of logistic regression and random forest models in predicting and classifying air quality levels and underlines the significance of effective AQI analysis for tackling air pollution and its effects on public health and the environment. After evaluating all models, the classification using the Apache Spark pipeline achieves an accuracy of over 0.96 and a runtime of only 3.54 seconds. The execution times of Logistic Regression (LR) and Random Forest (RF) vary significantly across platforms for Apache Spark pipelines, non-Apache Spark pipelines, and traditional machine learning.