MALACHITE—Enabling Users to Teach GUI-Aware Natural Language Interfaces

Users can adapt contemporary natural language interfaces (NLIs) by teaching the NLIs how to handle new natural language (NL) inputs. One promising approach is interactive task learning (ITL), which enables users to teach new NL inputs for multi-modal systems. While recent advances enable users to teach the syntactic and semantic level of the NL inputs through ITL, NLIs are still not able to learn how to consider the context, such as the current state of the graphical user interface (GUI). To address this challenge, we designed MALACHITE through three formative studies. MALACHITE enables users to successfully teach NL inputs on a semantic and syntactic level leveraging the GUI screen of a data visualization tool. With two evaluative studies, we provide evidence that with MALACHITE ’s suggestions, users significantly improve their accuracy by a factor of 2.3 in teaching GUI-dependent NL inputs in contrast to those without MALACHITE ’s suggestions.