Your Data, Your Model: A Framework for Training and Deploying Foundational Language Models for Embedded Devices

Large language models are being deployed for various applications, and they are being scaled to larger parameter sizes. However, this makes it challenging to use these models for embedded applications due to the limited computing memory and processing capabilities available on embedded platforms. Consequently, it limits the usage of these models to invocation through function calls to a remotely hosted platform, which introduces latency and privacy challenges. A smaller parameter-sized language model running locally on embedded platforms could mitigate this challenge. However, such smaller models make trade-offs with dataset diversity and parameters, leading to inaccurate prompt responses and high hallucination rates. In response, we present our ongoing work: OTTER, a framework for pre-training custom language models and fine-tuning language models for embedded applications. We observe that smaller parameter-sized models outperform larger ones when curated with appropriate pre-training and fine-tuning datasets. Their smaller size allows these models to perform inference on constrained embedded platforms. Building on this framework, we train a custom model for breathing detection and observe high accuracy.