Estimation of Speech Balloons with Unknown Words Based on Behavior of Reading Aloud English Comics

English learning is essential in the modern world and becomes more effective by estimating learners’ internal state. Then, we estimate the learners’ internal state by the learners’ behavior. In this study, we propose a method for estimating speech balloons including unknown words of English learners based on their behavior while reading English comics aloud. We believe that unknown words for learners show learners’ internal states such as confusion, and the use of comics makes learners keep their motivation as entertainment. We recorded the speech and eye gaze of learners reading comics aloud on PCs with a headset and an eye tracker and trained an SVM using 47 features calculated from the recorded data and the text information. We collected data from 40 Japanese university students and evaluated user-dependent and user-independent estimation, and confirmed that the proposed method performed better than the conventional method using only text information in both estimations. In addition, we confirmed that the user-independent estimation performed better than the user-dependent estimation. Moreover, we discussed the connection between effective features for the estimation and human behaviors when finding unknown words.