Phase space reorganization and traveling wave emergence driven by non-Kerr effects in nonparaxial optical media

In this article, the nonlinear Helmholtz equation with non-Kerr nonlinearity, such as self-steepening and self-frequency shift, is considered. A traveling wave transformation is applied, and an extended nonlinear Helmholtz equation is reduced to a Hamiltonian dynamical system. Then, the reduced Hamiltonian system is analyzed by the classification of equilibrium points, phase space analysis, and the construction of exact wave solutions. The relationship between the reduced dynamical coefficients and the original physical parameters is further established through a parameter space analysis. It is shown that self-steepening directly modifies the reduced dynamics, whereas self-frequency shift acts through the compatibility condition for the real traveling wave reduction. Together, these non-Kerr effects reshape the phase space geometry and traveling wave structure. Localized and periodic traveling waves are obtained, with their existence determined by the balance among dispersion, nonparaxiality, Kerr nonlinearity, and non-Kerr effects. Furthermore, a periodically forced version of the reduced system is examined to study the transition from regular to irregular dynamics. It has been observed that external forcing can induce complex oscillatory behavior. Bifurcation analysis, time series evolution, phase space, largest Lyapunov exponent, and Poincaré section demonstrate the emergence of quasiperiodic and chaotic responses under sufficiently strong forcing. All analytical branches are verified through full-equation residual evaluation, while few selected branches are additionally examined through direct numerical propagation and robustness tests under complex Gaussian perturbations. The results show that self-steepening directly renormalizes the effective nonlinear dynamics, whereas self-frequency shift restricts the admissible real-envelope traveling wave manifold.