Pairing Disparate Datasets for Generative Outputs Across Music and Architecture

Most multimodal models utilise semantically consistent data samples for model training and generative outputs. However, for design and composition work, semantic correlations are not the only means for cross-modal translations as suggested by synaesthetic phenomena. In this paper, we propose several methods to pair disparate datasets for training a model to translate between musical compositions and architectural design. We adopt and augment our own datasets to extract high-level features for both music and architectural samples. We utilise self-organising maps and agglomerative clustering strategies to pair data which is then used to train a conditional GAN. The techniques are evaluated through qualitative and quantitative assessment and are also applied in generating architectural compositions. Our experiments suggest promising initial results for feature pairing using hierarchical agglomerative clustering when generating images. We conclude by discussing the meaning and utilization of our proposed techniques as a means to systematically design with abstract features across modalities.