Integration of Explainable Artificial Intelligence and Multimodal Machine Learning for Drivers’ Fitness

In order to fully utilize the data collected from different sources and transform them into valuable assets for the prediction and explanation of Artificial Intelligence (AI) systems, this paper introduces an approach that combines Multimodal Machine Learning (MML) with Explainable AI (XAI). The goal is to provide insights into a deeper understanding of driver performance in terms of mental fatigue in drivers. The performances of the drivers can be assessed primarily based on their mental fatigue levels. Detecting mental fatigue using MML with explainability remains a challenge, especially when heterogeneous data are collected in an unsupervised manner. This paper includes multiple modalities in primary prediction and explanation tasks, enabling Multimodal XAI (MXAI). The work used vehicular telemetry data collected from multiple driving scenarios in Spain and Italy, providing a rich multi-source dataset for MML model development. Here, MML integrates information fusion, co-learning, and reasoning to analyse multivariate unlabelled data for fatigue detection. It applied k-means clustering on these unlabelled data, followed by classification using Random Forest and XGBoost, effectively creating a semi-supervised learning approach. In this study, XAI is used to enhance the transparency and interpretability of the MML model. Here, the contribution of various parameters to fatigue classification was examined using SHAP–Shapley Additive Explanations. Hence, the work contributes to driver fitness using MML to improve model accuracy and robustness, as well as XAI for model interpretability and transparency in detecting fatigue-related patterns.