LISP: Learning In-Camera Signal Processing for Lightweight Computer Vision

Computer vision (CV) has advanced significantly with learning-based approaches, like deep neural networks, which have reached human-level accuracy. However, implementing CV algorithms on Internet-of-Things (IoT) devices remains challenging due to latency and limited processing power. A common drawback of existing approaches is their inability to fully utilize the processing power available in camera-based systems, forcing them to rely on the low-power CPUs of IoT devices for all operations or the complex redesign of hardware architecture. To address this, we aim to offload most CV algorithm operations to the camera's hardware image signal processor (ISP). Developed for image classification tasks, our approach, LISP, learns proper ISP parameters to extract key features for classification from images. Then, the captured images can be easily classified by lightweight rules on the IoT device. We have evaluated LISP on three diverse image classification tasks using an off-the-shelf camera, demonstrating high accuracy, low runtime, and minimal memory usage,