NeuralCasting: A Front-End Compilation Infrastructure for Neural Networks

This study presents the development of NeuralCasting, a front-end compiler framework capable of converting (casting) neural networks encoded in the ONNX format to optimized C code. The primary objective is to enable the compilation of neural networks depending only on standard C libraries, thus eliminating the need for a separate inference engine, such as ONNX runtime. Furthermore, this feature allows the generation of C code suitable both for bare metal embedded devices and other resource-constrained devices, extending model applicability to a plethora of hardware targets. The framework addresses critical applications such as real-time audio processing, especially regarding latency constraints. As an example use case, we compile three different models: a Multi-Layer Perceptron (MLP) composed of fully connected layers, a ResNet cell for image recognition, and an NSNet 2 for speech enhancement and noise suppression. We analyze and compare the performance with other widely used frameworks, such as ONNX Runtime and PyTorch. Our findings indicate that the developed compiler successfully generates optimized C code that meets real-time processing requirements for latency-sensitive applications, outperforming ONNX Runtime and PyTorch and reaching a speedup close to $\mathbf{x} 10$ for small-sized MLP models, which are suitable for deployment on edge devices.