Teaching Things To Think: Bootstrapping Local Reasoning for Smart(er) Devices

Smart devices often use natural language understanding (NLU) to infer user goals. NLU works by extracting user intents from text, then mapping these intents to rigid sets of rules that specify system responses (i.e., slot filling). NLU behaves best when intents and their targets are well-specified (e.g., "set brightness to 50%"), and when the rules implement all desirable responses. NLU falters, however, when responses demand a deeper level of reasoning than predefined rules can provide, e.g., when intents are under-specified (e.g., "make a warm glow") or context-sensitive information is requested (e.g., "why is it so hot in here?"). Toward more flexible user-device interactions, we re-frame the challenge as one of building thoughtful things: devices that leverage a local capacity for reasoning about their internal states to generate responses rather than retrieving them from constrained rule sets. We propose an end-to-end framework for "teaching things to think" that trains a language model to generate valid, device-specific actions and natural language explanations in response to unconstrained language that is out-of-scope for slot filling NLU. Our key insight is that we can bootstrap this generative model from purely synthetic data constructed using a formal definition of a device’s state. The resulting model is small enough to run on-device, with no runtime cloud dependency. We implement two thoughtful things (a lamp and a thermostat) on real hardware and explore their potential to generate and explain device states in response to commands and inquiries.