Integrating Wireless and Sound Sensing with LLM for Accurate Keystroke Recognition

Remote and accurate keystroke recognition is essential for improving human-computer interaction and developing advanced analysis techniques. We propose a novel system that combines WiFi Channel State Information (CSI), acoustic signal analysis, and large language models (LLMs) to enhance keystroke recognition accuracy. While prior research has demonstrated the effectiveness of WiFi signals for detecting keystrokes by analyzing unique patterns in channel state information (CSI) variations, we extend this approach by incorporating acoustic data from keyboard strokes, as well as the contextual capabilities of LLMs. By combining two modalities (WiFi CSI and acoustic data), we increase the accuracy and robustness of keystroke detection. This dual-input approach combined with leveraging commercially available LLMs, not only detects and identifies keystrokes but also recognizes complete words and sentences. We have been able to enhance accuracy using WiFi CSI augmented with acoustic information to achieve keystroke recognition accuracy of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{8 1. 5 \%}$</tex>. With the inclusion of LLM in our system, our keystroke identification accuracy for sentences has increased to 99 %.