AttnPINNs: Physics-Informed Neural Networks under the Self-Attention Mechanism for Solving Partial Differential Equations

Abstract. Physics-informed neural networks (PINNs) have been widely applied in solving various physical models, including partial differential equations (PDEs). However, they have a considerable probability of failure when simulating dynamical systems with multiscale, high-frequency, or chaotic behaviors. The possible reason might be that the majority of PINNs methodologies regard space and time as a unified entity, thereby neglecting temporal dependencies across previous or subsequent time steps. In this paper, an advanced network structure, referred to as AttnPINNs, is proposed by stacking self-attention blocks behind a pretrained PINN. We introduce a sequence operator to transform the inputs from spatio-temporal points into sequential formats and a self-attention layer to capture the correlation of solution sequences. Furthermore, we provide a rigorous proof of convergence, indicating that merely the introduction of self-attention blocks is sufficient to bring about significant performance improvements by comparing the AttnPINNs with other advanced architectures (e.g., PINNs, QRes, first-layer sine, and PINNsFormer). Meanwhile, the numerical experiment results show that AttnPINNs method demonstrates superior performance and outperforms most of the other strategies on a wide range of PDEs, the solutions of which tend to have abrupt changes or exhibit multiscale, high-frequency, and chaotic properties.