Hierarchical Demand Based Resource Allocation for On-Device Inference

The use of wearables such as smartwatches, earbuds, and smart rings is expanding rapidly. During the COVID-19 pandemic, individuals increasingly relied on continuous vital signs monitoring, such as oxygen saturation (SpO2) and heart rate (HR), provided by wearables to seek timely medical assistance. As individual sensors become more affordable and feature-rich, advanced sensing capabilities that were once exclusive to high-end wearables are becoming widespread. These vital signs are often aggregates, derived from processing large amounts of data, which can strain the low-power micro-controllers typically used in wearables. While recent advancements have led to the development of low-power microcontrollers with integrated multi-core processors and accelerators, efficiently utilizing these resources, along with the capabilities of the advanced sensors, remains challenging. In this position paper, we propose a hierarchical resource management strategy to optimize system performance, focusing on two key factors: (i) the guarantees needed for specific computations, and (ii) the resource demands these computations impose. By extending operating system primitives and introducing a modular middleware framework, we argue that this approach can facilitate the seamless integration of complex applications, leveraging the capabilities of a hierarchy of devices. This strategy, we contend, can significantly improve the efficiency and robustness of critical sensing systems, providing a more adaptable, privacy-preserving, and resilient solution.