Hunting Cooperation: Its Impact in a modified May–Holling–Tanner model

Abstract In this work, a modified May–Holling–Tanner predator-prey model is analyzed, considering an alternative food source for predators and hunting cooperation between them. To describe the dynamics of the model, we demonstrate the existence of a positively invariant region, the boundedness, and permanence of the trajectories, and prove that the origin is a hyperbolic repeller. We provide necessary and sufficient conditions for the existence and explicit form of up to two positive equilibria. One equilibrium is always a hyperbolic saddle, while the other can be an attractor, repeller, or weak focus. Additionally, we found two key structures: (i) a separatrix curve on the phase plane dividing the behavior of trajectories into qualitatively distinct regions, and (ii) a homoclinic curve generated by the stable and unstable manifolds of a saddle point in the interior of the first quadrant. These structures highlight the system’s sensitivity to initial conditions, particularly near the separatrix. Bifurcations can occur in the system, including transcritical and Hopf bifurcations, which further influence the model’s dynamics. Finally, numerical simulations are presented to validate the analytical results and illustrate that hunting cooperation is unfavorable for the coexistence of two species when the strength of hunting cooperation increases.