Toward Next-Generation Human–Computer Interface Based on Earables

We introduce a new type of human–computer interface—mobile acoustic field (MAF). MAF explores a variety of onboard sensors, such as speaker transducers, inertial measurement unit (IMU), and feedforward and feedback microphones, on earphones/earbuds to capture a wide range of acoustic activities both inside-out and outside-in. We anticipate that the exponential advancement of generative AI, powered by the sheer size of cloud computing resources, will enable personal agents to sense, understand, and further interact with this MAF, offering many exciting mobile services to users. In this article, we summarize our preliminary results, discuss the downstream applications, technical challenges, and our vision on this exciting research field.