Multimodal Sensing and Machine Learning to Compare Printed and Verbal Assembly Instructions Delivered by a Social Robot

In this paper, we compare a manual assembly task communicated to workers using both printed and robot-delivered instructions. The comparison was made using physiological signals (blood volume pulse (BVP) and electrodermal activity (EDA)) collected from individuals during an experimental study. In addition, we also collected responses using the NASA Task Load Index (TLX) survey. Furthermore, we mapped the collected physiological signals to the responses of participants for NASA TLX to predict their workload. For both classification problems, we compared the performance of Convolutional Neural Networks (CNNs) and Long-Short-Term Memory (LSTM) models. Results show that for our CNN-based approach using multimodal data including both BVP and EDA gave better results than using just BVP (approx. 8.38% better) and EDA (approx. 20.49% better). Furthermore, our LSTM-based approach also had better results when we used multimodal data (approx. 8.38% better than just BVP and 6.70% better than just EDA). Overall, CNNs performed 7.72% better than LSTMs for classifying physiologies for paper vs robot-based instruction. The CNN-based model also provided better classification results (approx. 17.83% better on an average across all responses of the NASA TLX) within a few minutes of training compared to the LSTM-based models.