Multimodal journaling as engagement tool for parents of premature babies

Parents of premature babies face extreme psychological challenges when a child is born too early - a situation for which no parents can be prepared. Parental engagement solutions, such as logging the baby's progress, can alleviate this stress by fostering a comforting feeling of involvement in the care and a clearer understanding of the overall situation. This paper presents a multimodal journaling prototype designed specifically for parents of premature babies in the Neonatal Intensive Care Unit (NICU). The general goal was to test its feasibility and applicability as an engagement tool, while the specific goal was to evaluate user interaction methods from both technical performance and usability perspectives. The implementation, built around a multimodal integration module, demonstrated high technical performance, yielding greater than 95% accuracy. However, usability evaluation revealed a significant finding: observed interaction patterns showed a substantially larger proportion of speech-only interaction compared to the intended multimodal combination of speech and gesture. To fully realize the benefits of a multimodal approach for parental engagement, and more broadly for patient engagement in digital health, further research is imperative. This work must focus on refining the interaction design to better align with the lived parental experience and seamlessly support natural multimodal usage.