An explainable cockpit speech emotion recognition model towards affective human-AI interaction in aviation

Aeronautical decision-making by pilots and air traffic controllers (ATCOs) is often complex and requires integrating multiple pieces of information from the environment. It necessitates emotional regulation to make the appropriate aeronautical decision based on the information perceived from the scenario. Hence, recognising the emotions of pilots and ATCOs is a prerequisite to designing appropriate support for emotion-aware flight operations. Nevertheless, generic speech emotion recognition models might face challenges in recognising emotions from aviation communications with a specific phraseology. In this regard, we first introduce the Cranfield Aviation Speech Emotion Set (CASES), a corpus tailored for speech emotion recognition in aviation. This corpus comprises 30 diverse utterances recorded from 20 pilots and ATCOs with multiple accents across seven basic emotions. We then developed a deep learning model based on long short-term memory (LSTM) and CASES to recognise emotions in aviation communications. A data augmentation technique has been applied to enhance the model's prediction performance. Our proposed model achieves a test accuracy of 84.52 %, outperforming models trained with generic datasets and demonstrating the need for a tailor-made corpus in aviation applications. Shapley additive explanations (SHAP) are further leveraged to reveal feature importance in the emotion recognition process, thereby enhancing the model’s explainability. Overall, the model supports trustworthy emotion-aware flight operations for minimising emotional impacts on aeronautical decision-making.