PONIFY: Pose-Driven Painting Sonification to Enhance Augmented Artwork Perception through Sense of Dynamics

Integrating auditory elements that complement the visual features of the paintings can enhance artwork appreciation. <i>Image sonification</i> has advanced this paradigm by automatically generating matching sounds. However, existing techniques rely on basic features such as color without integrating complex contextual information. Based on the insights from our formative study, we introduce <i>Ponify</i>, a sonification method that translates a painting’s sense of dynamics into music through pose analysis. Ponify analyzes the limb movements of human figures in paintings and configures musical parameters such as the <i>tempo</i> and <i>density</i> to convey dynamic perceptions. Evaluation studies indicate that Ponify generates music that aligns harmoniously with the artwork, surpassing the performance of methods without pose analysis. In addition, Ponify enhances viewers’ enjoyment and empathy without reducing their concentration. We conclude that incorporating pose-based dynamics into sonification can offer a more immersive and emotionally engaging art appreciation experience, opening new possibilities for multimedia interpretation.