Assessing Usefulness, Ease of Use, and Recognition Performance of Semi-Automatic Mulsemedia Authoring

Mulsemedia (Multiple Sensorial Media) authoring poses a considerable challenge as authors navigate the intricate task of identifying moments to activate sensory effects within multimedia content. A novel proposal is to integrate content recognition algorithms that use machine learning (ML) into authoring tools to alleviate the authoring effort. As author subjectivity is very important, it is imperative to allow users to define which sensory effects should be automatically extracted. This article conducts a twofold evaluation of the proposed semi-automatic authoring. The first is from a user perspective within the STEVE 2.0 mulsemedia authoring tool, employing the Goal–Question–Metric (GQM) methodology and a user feedback questionnaire. Our user evaluation indicates that users perceive the semi-automatic authoring approach as a positive enhancement to the authoring process. The second evaluation targets sensory effect recognition using two different content recognition modules, quantifying their automatic recognition capabilities against manual authoring. Metrics such as precision, recall, and F1 scores provide insights into the strengths and nuances of each module. Differences in label assignments underscore the need for ML module result combination methodologies. These evaluations contribute to a comprehensive understanding of the effectiveness of sensory effect recognition modules in enhancing mulsemedia content authoring.