Musical Scene Detection in Comics: Comparing Perception of Humans and GPT-4

This paper aims to detect musical scenes for improving background music generation for comics. Musical scenes can be defined as scenes where background music is enhancing the media experience. A significant barrier in detecting such scenes for comics is the absence of a ground truth to compare against. This makes our work one of the first in this field to address this task. Hence, we first analyse musical scenes through the dialogues in anime films adapted from Japanese comics (manga). Through our analysis we discover that, apart from the existing literature on dimensions of change in scene segmentation, musical scenes are also triggered by emotions. We then build an internal dataset to engineer prompts for GPT-4 to recognise musical scenes. The results of the generated outcomes are evaluated against the internal dataset as well as human evaluation. We are able to prove that prompt engineering can objectively improve the musical scene detection. Our results indicate that GPT-4 has 62.5% agreement rate with human evaluators. The implications of this research on background music generation are relevant for various media. This paper invites further investigation on this topic.