Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms "Typo" Correction

The large integration of microphones into devices increasesthe opportunities for Acoustic Side-Channel Attacks (ASCAs), as these can be used to capture keystrokes’ audio signals that might reveal sensitive information. However, thecurrent State-Of-The-Art (SOTA) models for ASCAs, including Convolutional Neural Networks (CNNs) and hybrid models, such as CoAtNet, still exhibit limited robustness underrealistic noisy conditions. Solving this problem requires either: (i) an increased model’s capacity to infer contextualinformation from longer sequences, allowing the model tolearn that an initially noisily typed word is the same as afuturely collected non-noisy word, or (ii) an approach to fixmisidentified information from the contexts, as one does nottype random words, but the ones that best fit the conversationcontext. In this paper, we demonstrate that both strategiesare viable and complementary solutions for making ASCAspractical. We observed that no existing solution leveragesadvanced transformer architectures’ power for these tasks andpropose that: (i) Visual Transformers (VTs) are the candidatesolutions for capturing long-term contextual information and(ii) transformer-powered Large Language Models (LLMs) arethe candidate solutions to fix the “typos” (mispredictions) themodel might make. Thus, we here present the first-of-its-kindapproach that integrates VTs and LLMs for ASCAs.We first show that VTs achieve SOTA performance inclassifying keystrokes when compared to the previous CNNbenchmark. Second, we demonstrate that LLMs can mitigatethe impact of real-world noise. Evaluations on the natural sentences revealed that: (i) incorporating LLMs (e.g., GPT-4o) inour ASCA pipeline boosts the performance of error-correctiontasks; and (ii) the comparable performance can be attained bya lightweight, fine-tuned smaller LLM (67 times smaller thanGPT-4o), using Low-Rank Adaptation (LoRA). Our resultsand findings highlight the practical viability of our solutiontoward effective ASCA.