Limit cycles near a nilpotent center and a homoclinic loop to a nilpotent singularity of Hamiltonian systems

Abstract For a planar analytic Hamiltonian system, which has a period annulus limited by a nilpotent center and a homoclinic loop to a nilpotent singularity, we study its analytic perturbation to obtain the number of limit cycles bifurcated from the periodic orbits inside the period annulus. By characterizing the coefficients and their properties of the high-order terms in the expansion of the first-order Melnikov function near the loop, we provide a new way to find more limit cycles. Moreover, we apply these general results to concrete systems, for instance, an $(m+1)$ th-order generalized Liénard system, and an m th-order near-Hamiltonian system with a hyperelliptic Hamiltonian of degree $6$ .