Improving Continuous Emotion Annotation in Video Platforms via Physiological Response Profiling

Many video applications (e.g., gaming, meeting, tutoring) aim to improve the user's interaction experience based on continuously inferred user emotion. To infer user emotion, these apps typically deploy machine learning models, trained with continuously collected emotion ground truth labels. However, as continuous annotations are generally collected as emotion self-reports during video consumption (using an auxiliary device), they incur significant annotation effort. To address this problem, we propose PResUP, a framework that creates users' profile using physiological responses (e.g., GSR or galvanic skin re-sponse) and deploys an LSTM network to identify the opportune probing moments for emotion ground truth (emotion self-report) collection instead of continuous annotation. We evaluate the proposed approach on a large-scale publicly available dataset (CASE) containing the physiological signals of subjects during video consumption. The evaluation of PRes UP reveals that it reduces the probing rate by 30.3 % (on average), detects the opportune probing moments with a TPR (True Positive Rate) of 86.1 %, and yet maintains the quality of the emotion annotations as observed in the continuous self-reports. Furthermore, we evaluated the generalizability of PResUP on another public dataset (K-emocon); which reveals an average probing rate reduction of 25.71 %. These results underscore the efficiency of PResUP in reducing the continuous emotion annotation overhead.