A Topologically Informed Unified Adaptive Multi-Modal Data Fusion Design for Automatic Target Recognition

Existing automatic target recognition (ATR) paradigms leverage multi-modal data extensively. Data from various sensory modalities can improve accuracy providing unique information and enhanced reliability through the mitigation of individual modality failures. However, challenges in multi-modal data fusion persist since it demands modality-specific knowledge for effective feature extraction. Furthermore, modality-specific feature engineering increases system complexity and reduces reliability. When the number of modalities, their respective collection devices, and individual device channels deployed becomes voluminous, data fusion solutions prohibit scalability. This paper introduces a unified adaptive multi-modal data fusion framework for ATR. First, it leverages topological data analysis (TDA), particularly persistent homology, to extract topological features from multi-modal data. These features are then fused for ingestion into machine learning (ML) models, such as support vector machines (SVM) and deep neural networks (DNN). Second, the framework uses conformal prediction scores derived from TDA features to evaluate the ML model's certainty on a given ATR task. Depending on the ML model's certainty, this paper develops an adaptive multi-modal data fusion solution. Five sensory modalities are considered, including electro-optical (EO), infrared (IR), acoustic, passive radio frequency (pRF), and seismic, which are extracted from the US Air Force Research Laboratory's Information Directorate (AFRL/RI) 2021 ESCAPE II multi-modal dataset. In the associated ATR ML tasks involving seven distinct ESCAPE II target vehicles, the results demonstrate that the proposed solution/framework is computationally viable and lightweight in terms of the number of parameters, improving robustness while maintaining high accuracy in the presence of modality uncertainty and reduced modality quality. In effect, the proposed framework shows promise in supporting robust, high-accuracy ATR and continuous target custody within US DoD agency mission spaces, as well as broad applicability to civilian multi-modal sensor datadriven autonomy-based industry/monitoring capabilities.