A Comparison of Late-Fusion Training Strategies for Quad-Modal Joint Embeddings

Multimodal methods that align the semantics of two data modalities-usually text and image-have become increasingly widespread in recent years. While such bi-modal alignments have received much attention, less work has been done toward aligning more than two modalities. This paper explores several strategies for training late-fusion joint-embedding models with up to four simultaneous modalities. We show that, even on a comparatively small dataset, there are clear and disproportional advantages when considering all modalities concurrently rather than consecutively. The results indicate that, when aligning multiple modalities, a training set with even weak alignment between all modalities substantially outperforms the use of several pair-wise alignments and that the addition of a modality can disproportionally increase alignment quality across all modalities.