The Case for Micro Foundation Models to Support Robust Edge Intelligence

This paper advocates the concept of micro foundation models (μFMs), recently introduced by the authors to describe a category of self-supervised pre-training solutions that we argue are necessary to support robust intelligent inference tasks in Internet of Things (IoT) applications. The work is motivated by the fact that collecting sufficient amounts of labeled data in IoT applications to train AI/ML tasks is challenging due to the difficulties in labeling such data after the fact. In the absence of sufficient labeled data, supervised training solutions become brittle and prone to overfitting. Self-supervised training obviates the collection of labeled data, allowing pre-training with the more readily available unlabeled data instead. Specifically, the μFMs discussed in this paper use self-supervised pre-training to develop an encoder that maps input data into a semantically-organized latent representation in a manner agnostic to the downstream inference task. Our preliminary work shows that this (unsupervised) encoder can be moderately sized, yet produce a latent representation that simultaneously supports the fine-tuning of multiple downstream inference tasks, each at a minimal labeling cost. We demonstrate the efficacy of this pre-training/fine-tuning pipeline using a vibration-based μFM as a running case study. The study shows that the fine-tuning of inference tasks on top of the aforementioned encoder-produced latent representation needs orders of magnitude fewer labels than supervised training solutions, and that the resulting tasks are significantly more robust to environmental changes and easier to adapt to domain shifts compared to their supervised counterparts. Furthermore, we show that inference algorithms based on our example μFM can be executed in real time on a Raspberry Pi device, making the approach viable for the IoT space. We conclude that μFMs are a preferred (and likely necessary) route to support robust intelligent sensing on IoT devices in subareas where labeled data collection is challenging. The paper is a call for the research community to invest in μFM research for IoT applications.