Machine Learning Enabled Data Quality (DQ) Assessment Framework for connected vehicles data

Connected vehicles leverage innovations in sensors, IoT, cloud computing, AI, and 4G/5G to produce real-time vehicle data, enhancing applications in navigation, fleet management, diagnostics, and maintenance; improving cost-efficiency, revenue, customer satisfaction, and safety. However, maintaining data quality in connected vehicles is challenging. Classical data quality assessment frameworks are inadequate for the complexity of connected vehicles, necessitating improved methods for assessing data quality in this domain. This research integrates machine learning and statistical methods with classical frameworks to enhance data quality assessment. A literature review identifies data quality challenges, existing frameworks, their strengths and limitations. Implementing a classical framework with real-world connected vehicle data uncovers issues like missing, delayed, and invalid data but fails to answer some data quality requirements, which are identified as gaps leading to the development of three scenarios to leverage machine learning. Scenario I uses logistic regression to detect non-communicating vehicles addressing delayed and missing data issues.Scenario II forecasts missing mileage using a time series method. Scenario III assesses data accuracy using Light Gradient-Boosting Machine and Random Forest. The implementation of these scenarios provided promising results. Scenario I detects noncommunicatingvehicles with F1-score of 0.85. Scenario II forecasts missing mileage with lower RMSE compared to state-of-the-art methods. Scenario III detects inaccurate fuel consumption with 97% accuracy and F1-score of 0.78, outperforming Isolation Forest. In conclusion, implementing a classical data quality assessment framework with real-life vehicle data highlights various data quality issues and reveals certain limitations. Machine learning and statistical methods help to address these limitations. Therefore, a new framework that integrates classical data quality assessment with machine learning for connected vehicles data is proposed.