Stepwise AI Interpretive Approach for Multimodal Data Fusion

In recent years, Artificial Intelligence technology has excelled in various tasks and is taking the world by storm. However, the various transformations in neural networks make it difficult to make sense of the reasons why decisions are made. For this reason, trustworthy AI techniques have started gaining popularity. AI interpretability serves as an anchor point in the field of data fusion for multimodal AI, providing in-depth insights. The paper proposed a Stepwise AI Interpretative (SAII) approach using different pairing methods of ‘one-to-one’ and ‘many-to-many’ in an attempt to illustrate/demonstrate the interpretability of the process of pairing images and text. A counterfactual instantiation method was used to compare the whole-local relationship between a set of images and their associated descriptive text. The approach was evaluated via ‘task performance’.