Retrieval Augmented Generation with Multi-Modal LLM Framework for Wireless Environments

Future wireless networks aims to deliver high data rates and lower power consumption while ensuring seamless connectivity, necessitating robust wireless network optimization. To achieve this, wireless network optimization is necessary. Large language models (LLMs) have been deployed for generalized optimization scenarios. To take advantage of generative AI (GAI) models, retrieval augmented generation (RAG) is proposed for multi-sensor wireless environment perception. Utilizing domain-specific prompt engineering, we apply retrieval-augmented generation (RAG) to efficiently harness multimodal data inputs from sensors in a wireless environment. Wireless environment perception is necessary for global LLM optimization tasks. Key pre-processing pipelines including image-to-text conversion, object detection, and distance calculations for multimodal RAG input from multi-sensor data from different devices are proposed in this paper to obtain a unified vector database crucial for optimizing large language models (LLMs) in global wireless tasks. Our evaluation, conducted with OpenAI's GPT and Google's Gemini models, demonstrates an 8%, 8%, 10%, 7%, and 12% improvement in relevancy, faithfulness, completeness, similarity, and accuracy, respectively, compared to conventional LLM-based designs. Furthermore, our RAG-based LLM framework with vectorized databases are computationally efficient providing real time convergence under latency constraints. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>