Dynamic behavior and modulation instability of the generalized coupled fractional nonlinear Helmholtz equation with cubic–quintic term

Abstract In this work, the main objective is to study the dynamic behavior of the generalized coupled fractional nonlinear Helmholtz equation with cubic–quintic term, which describes soliton propagation in nonlinear optics through the theory of dynamical system. First, the original equation is transformed into an integer-order coupled partial differential equation using Atangana’s fractional derivative. Then, considering traveling wave and linear transformations, a two-dimensional planar dynamical system is constructed. Through phase portraits, stability analysis, and parameter sensitivity analysis, the dynamical behavior of the system is studied. Next, considering the dynamic behavior of the system under triangular periodic perturbation and logarithmic perturbation respectively, the reasons for the evolution of the system behavior patterns under different initial conditions are analyzed. Through qualitative analysis, we have avoided the limitations and errors of precise solution methods, and obtained the stability of the system equilibrium point under parameter changes, as well as the dynamic behavior of the system, including periodic and chaotic behaviors. Finally, by investigating the modulation instability of the system, the conditions for maintaining stability of the system are obtained through the linear stability analysis method.