Multimodal data generative fusion method for complex system health condition estimation

For the health management of complex systems, the high value of such systems often necessitates multimodal monitoring data, including video surveillance, internal sensors, empirical formulas, and even digital twins. Therefore, it is essential to design an effective intelligent fusion method for multimodal data. Firstly, a global monotonicity calculation method and a time series data augmentation technique are developed to address the inconsistencies arising from varying temporal lengths across different modalities. Secondly, in response to the need for efficient time series fusion, we propose a fast sequential learning network architecture along with a time series generative data structure. Finally, we introduce a many-to-many transfer training approach that culminates in the formation of a Multi-source Generative Adversarial Network (Ms-GAN). Numerical experiments and monitoring datasets are employed to validate the effectiveness of this multimodal generative fusion method. Notably, Ms-GAN enhances traditional GANs-typically limited to learning single data distributions-by enabling multimodal data fusion capabilities. This advancement holds significant promise for applications in various fields such as multimedia processing and medical diagnosis.