Neural networks for nonlinear system identification

From real-time control of industrial robots to precise prediction of electric vehicle battery range, accurate models of nonlinear dynamic systems are essential for modern technologies. While first-principles modeling often struggles with complex systems due to unknown effects and required expertise, Nonlinear System Identification (NLSID) offers a data-driven alternative. However, NLSID faces challenges regarding model performance, speed, structures, and training. Deep learning has shown remarkable success in handling complex problems across various domains. Yet, despite previous contributions in Neural Network-based System Identification (NNSID), a comprehensive deep learning perspective on addressing these challenges remains unexplored. This thesis investigates how established deep learning practices can be adapted and how novel NNSID model structures can be developed to address the identified challenges of NLSID for discrete-time, finite-dimensional dynamic systems. The contributions of this thesis are fourfold: (1) We introduce a comprehensive training regime that adapts best practices from deep learning that are rarely or never used in NNSID, like advanced optimizers, regularization techniques, and systematic hyperparameter optimization, to improve the accuracy of existing NNSID model structures. (2) We develop non-autoregressive model structures for simulation and multi-step ahead prediction that minimize sequential dependencies to speed up training and inference. (3) We propose methods to incorporate prior knowledge into neural networks with minimal domain expertise through task-specific loss functions and data augmentation to improve model accuracy. (4) We develop techniques to model system variations, such as measurement errors and varying sampling rates through data augmentation, to improve generalizability. Extensive experiments validate these contributions: The training regime (1) significantly improves the accuracy of NNSID models on public NLSID benchmarks using identical model structures. The non-autoregressive model structures (2) achieve up to 100 times faster training and inference than conventional autoregressive model structures while maintaining or improving accuracy. In a comprehensive case study of inertial attitude estimation, the proposed incorporation of prior knowledge (3) enables our models to outperform established first-principles models consistently. Through modeling system variations (4), a single trained model achieves, without system-specific tuning, superior accuracy across systems with different motion patterns, sampling rates, and sensors compared to first-principles models. These results demonstrate that neural networks can indeed address the challenges of NLSID regarding model performance, speed, structures, and training. The thesis' contributions are combined into a general workflow for NNSID, from initial black-box modeling to progressive incorporation of prior knowledge. This workflow is implemented in the software library tsfast to enable the rapid development of accurate models with minimal prior knowledge and human intervention. For example, in industrial robotics, fast and accurate models can improve real-time control, potentially increasing manufacturing speed while maintaining precision. Similarly, for electric vehicles, accurate and generalizable models can enhance range prediction by adapting to individual battery characteristics and varying environmental conditions. While other challenges remain, these contributions could accelerate the adoption of data-driven modeling, potentially leading to more efficient and innovative technologies.