Communicating the dialog between painting and music with a mobile application

Contemporary visual artists often collaborate with musicians and composers to enrich their artworks and provide multisensory viewing experiences by leveraging the power of digital technology. In this paper, we present our approach to follow in their footsteps by creating a mobile application that provides “mixed-media interpretations” for a variety of visual artworks and music pieces, spanning over different time eras, with the objective to reflect the creative dialog and multifaceted interaction between their creators through time. Targeting the broader public, we attempt to capture a variety of interconnections between works of visual arts and music that may be of interest and are comprehensible to people with varying levels of knowledge on art or music, including not only inspirational but also personal and social relationships among the artists. We provide a six-layered conceptual model to ease the description of the content produced to that end while reporting several concrete examples. Then, we focus on how to communicate and digitally mediate the intertwined and often intricate interconnections in a concise way, proposing a “three-act” story scheme to progressively display short narratives on screen, combined with music reproduction on the mobile device. Finally, we explore the potential of the proposed approach through a user study with 15 participants and report a series of promising key findings, whereas also discussing future directions and pointing out open issues.