D4.1 Report on robustness and resilience MML with open-source models and database (short version)

This report presents the research and development efforts conducted focusing on the automation of the remote digital tower (RDT) applications, with multimodal machine learning (MML) through the development of robust and resilient machine learning (ML) models within the framework of Trusted Intelligent Systems. The Multimodal Machine Learning (MML) approach addresses translation, alignment, and co-learning through object detection from video and audio-text transcription for semantic analysis. For proof of concept, we developed an MML AI system for RDT applications, specifically for enhanced runway and taxiway monitoring and adaptive weather management. The development employed deep learning libraries such as YOLOv10 and YOLO-World for object detection, while audio transcription utilised WhisperATC with datasets from ATCO2 and LiveATC.net. Additionally, it explored out-of-distribution generalisation, conformal prediction, and model calibration to improve reliability and resilience. Finally, this work contributes to developing open-source AI models and datasets for intelligent air traffic monitoring, ensuring consistent performance under diverse conditions while fostering trustworthiness in AI-driven decision-making.