Unified Framework for Procedural Task Assistants powered by Human Activity Recognition

Context awareness is key to developing intelligent voice assistants that offer situated support for users performing various daily tasks, like cooking and machine use. Human Activity Recognition (HAR) with various sensors can be a powerful approach to provide assistants with user context, i.e., what they are doing within the task procedure. This workshop paper introduces PrISM---Procedural Interaction from Sensing Module---, a framework tool to develop and evaluate assistants for procedural tasks that utilize the context gained through HAR. The framework consists of several modules: data collection, HAR, postprocessing HAR outputs with task knowledge represented as a graph structure, and situated interactions such as step reminder and question answering. The framework is developed to be generalizable to different procedural tasks and input sensor sets (e.g., smartwatch, wearable camera, ambient sensor, etc). Developers and designers can use the framework to register a new task, collect an initial dataset, gauge expected accuracy regarding step tracking and interactions, and deploy the pipeline. We demonstrate example use cases in the cooking task and discuss future work.