A scalable RISC-V Coprocessor for Liquid Time-Constant Neural Network

Liquid Time-Constant recurrent neural networks (LTC) have shown promising results when processing nonstationary time-series data in applications like autonomous driving and navigation. In such applications, the ability of LTC networks to cope with unseen environments sets them apart from other neural network models. Their compact model architecture and high sparsity in principle also make LTC networks a viable candidate for edge applications. However, achieving high throughput and low latency on deeply embedded devices with soft-CPUs and thus limited compute power remains challenging for networks of reasonable size. This paper describes the design of a tightly-coupled RISC-V coprocessor for the acceleration of LTC inference in small- to medium-size FPGAs. The accelerator design takes into account profiling results of an optimized LTC software implementation. Exploiting sparsity in the LTC network results in a hardwareefficient design that scales well for any size of network. The coprocessor is fully configurable with respect to the trade-off between hardware resource utilization and acceleration speed. The resulting coprocessor implementation is verified and the speedup evaluated for different model sizes through simulation. Results show that, compared to an optimized software implementation, significant speedups of up to more than 100 times can be achieved. This speedup comes with little impact on model accuracy and low hardware resource utilization.