Towards Offline GenAI Fine Tuning Model with LoRA Derivatives for IoT Edge Server

The Internet of Things (IoT) has become increasingly pervasive, connecting a vast network of devices to collect and analyze data. However, the reliance on continuous internet connectivity poses challenges in regions with limited or unstable access. This paper investigates the feasibility of deploying offline Generative AI (GenAI) models on IoT edge servers, enabling autonomous data generation in disconnected environments. A significant challenge arises from the resource constraints inherent to edge devices, which often lack the computational power required to run sophisticated AI models. To address this, techniques such as model compression and quantization are considered to reduce the size and computational demands of the models, while maintaining acceptable accuracy. One such technique, Low-Rank Adaptation (LoRA), is examined in this study alongside its various derivatives. The primary contribution of this paper is a comparative analysis of several LoRA derivatives, including Quantized LoRA (QLoRA), Multi-task LoRA (MT-LoRA), and Adaptive LoRA (AdaLoRA), in fine-tuning the LLaMA 3.1 large language model (LLM) for IoT applications. The evaluation focuses on memory optimization and model performance, with experiments conducted using the 4-bit quantized version of LLaMA 3.1 8B. These efforts aim to create realistic simulation environments for testing and evaluating IoT systems under different conditions.