Dynamic behavior and modulation instability for a generalized nonlinear Schrödinger equation with nonlocal nonlinearity

Abstract This paper focus on the generalized nonlinear Schrödinger equation with triple refractive index and nonlocal nonlinearity, which is used to describe the evolution state of optical solitons in fiber optic propagation. Firstly, the complex exponential traveling wave transformation and substitution method are utilized to convert the equation into two-dimensional planar dynamic system, and then by phase portrait the developmental patterns of solutions are qualitatively analyzing. Meanwhile, the evolution of solutions under different disturbances in the system is discussed. Qualitative analysis of the system are displayed through Poincaré section and sensitivity analysis. Finally, the instability of the system is modulated by using linear stability analysis method, and the corresponding conditions for steady-state solutions and the gain spectrum function are obtained.