Physics-Informed Neural Networks for Approximating Loss Evolution of an Artificial Neural Network: Novel Approach to Implicit Regularization

The introduction of Physics-informed Neural Networks (PINNs) has catalyzed research into efficient machine learning methods that model the behavior of dynamical systems by embedding physical laws as prior knowledge. Additionally, as per the Universal Approximation Theorem, Artificial Neural Networks themselves are functions that approximate, with some arbitrary error, a mapping between an input and output. Treating Artificial Neural Networks as physical systems, we present a physics-informed neural network framework capable of approximating the loss evolution of another feedforward Neural Network (NN). Particularly, we embed partial differential equation terms characteristic of the NN learning process into the PINN loss function, which introduces a regularization constraint that allows the PINN to represent NN loss. In our experiments, we train varied PINNs to approximate, with arbitrary error, the loss evolution of a simple MNIST model. Altering parameters and partial differential equation terms supplies insight into the strengths and limitations of this physics-based method for representing the learning process of neural networks, enabling future research into loss forecasting methods capable of boosting NN efficacy.