Methodologies for Improved Optimisation of the Derivative Order and Neural Network Parameters in Neural FDE Models

This work presents and compares different methodologies for the joint optimisation of the fractional derivative order and the parameters of the right-hand-side neural network in Neural Fractional Differential Equation models. The proposed strategies aim to tackle the training difficulties typically encountered when learning the fractional order α together with the network weights. One approach is based on regulating the gradient magnitude of the loss function with respect to α, encouraging more stable and effective updates. Another strategy introduces an online pre-training scheme, where the network parameters are initially optimised over progressively longer time intervals, while α is updated more conservatively using the full time trajectory. The study focuses only on a foundational setting with one-dimensional problems, and numerical experiments demonstrate that the proposed techniques improve both training stability and accuracy. Nonetheless, the issue of non-uniqueness in the optimal derivative order remains, particularly in less well-posed scenarios, suggesting the need for further research in data-driven modelling of fractional-order systems.