Unsupervised physics-informed disentanglement of multimodal materials data

Materials, and the processes used in their synthesis, are commonly evaluated via a variety of experimental modalities, each individually describing some aspect of the process conditions, material structure , chemistry, material properties, and associated performance. Traditionally, materials experts are called upon to make sense of this collage of disparate information. However, emerging algorithms offer an opportunity to fuse these complementary measurements from multiple distinct modalities into a holistic, high-dimensional descriptor of the material state. We present herein a probabilistic framework for discovering such shared information in multimodal datasets. Through an unsupervised approach based on variational inference, we identify semantic clusters that encode correlation across modalities, thereby obtaining physically meaningful fingerprints that indicate distinct mechanistic regimes of performance. This multimodality can be further enriched through the incorporation of physics-based constitutive models that can facilitate cluster disentanglement. The probabilistic underpinnings of the approach provide uncertainty quantification to evaluate reliability of cross-modal estimation and quantify how individual modalities contribute more than the sum of their parts. The approach is demonstrated for a collection of three multimodal datasets related to material reliability.