From Pixels to Physicality: Wave-Mediated Interaction above a Screen

Abstract Visuo-physical displays that integrate visual information with physical interaction generally require additional actuation systems. Here, we propose a transparent visuo-physical panel that extends the role of a display front panel beyond its optical function to include acoustic wave propagation and radiation, enabling focused ultrasound above the screen. Unlike conventional mid-air interaction systems composed of hundreds of opaque ultrasonic transducers positioned separately from the display, the panel itself radiates ultrasound into air without obstructing the viewing area. Although waves excited at the panel boundary undergo complex propagation, dispersion, and reflection within the panel before radiating into air, we develop spatiotemporal inverse filtering to focus the radiated ultrasound at a prescribed target point in air. The focal position can be updated more than 100 times per second, and multiple focal points can be generated simultaneously. Acoustic pressures approaching 1 kPa enable object lifting, mid-air haptic feedback, and selective candle extinguishing, demonstrating physical interaction generated directly by the display panel.