Automation Surprise Detection Based on Facial Expression Changes and Operation Input Values

While driving support systems for electric wheelchairs are being developed to enhance safety in aging societies, they can cause automation surprise. This phenomenon, in which the system's unintended behavior causes discomfort or a sense of burden for the driver, undermines user trust. For the development of truly human‐centered systems, it is therefore essential to quantitatively evaluate whether a given support system induces such surprises and to provide feedback for its design. To enable this evaluation, this paper proposes a novel dual‐modal method for detecting automation surprises by integrating facial expression changes and operation input values. This approach enhances detection robustness by simultaneously analyzing the driver's affective responses and corrective actions. To evaluate our method, we conducted experiments with 13 drivers using a custom Unity‐based racing game designed to intentionally induce automation surprises. The experimental results confirmed that our integrated approach significantly reduced false positives and improved the F‐measure to 0.525, outperforming methods based solely on facial expressions (0.481) or operation inputs (0.516). Furthermore, the method achieved a recall of 0.615 in sections where drivers reported strong surprise, demonstrating higher accuracy compared to sections with weak surprise (0.500). © 2025 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.