A Novel Multimodal LLM-Driven RF Sensing Method for Human Activity Recognition

Human activity recognition (HAR) using radio frequency (RF) sensing has attracted significant attention due to its unobtrusive and privacy-preserving nature. Traditional HAR methods rely on task-specific deep neural networks trained on large labeled datasets, which can be time-consuming and resource-intensive. To address these challenges, we propose a novel approach that leverages multimodal large language models (MLLMs) for RF-based HAR. Specifically, we fine-tune Florence-2, a pre-trained vision-language model (VLM), on RF spectrogram data from the open-source Xethru Radar dataset. Our approach frames activity detection as a question-answering task, allowing the model to associate radar spectrogram features with specific activity classes through prompt-based interactions. Testing on three distinct activities (sitting, bending, and crawling), our fine-tuned model achieves 98% classification accuracy with minimal misclassifications. This work demonstrates the effectiveness of integrating VLMs with RF sensing data for scalable and adaptive HAR applications, opening new research directions for unified, prompt-based models in complex multimodal sensing tasks.