Food chain model: cooperative behavior in top predator and its impact on prey and middle predator demography with fear effect

Purpose This study aims to propose modified functional responses for both middle and top predator species to incorporate the observation of Suraci et al. [Nat. Commun., 2016, 7: 10698] due to antipredator behavior of the middle predator, and study how the fear effect in both prey and middle predator species, and hunting cooperation among top predator population influence the dynamics of a food chain model. Design/methodology/approach This study provides a qualitative investigation of a food chain model with modified Holling Type-II functional responses for middle and top predators. The fear effect in both prey and middle predator species, and the cooperative hunting behavior among top predator population is explored in this work. The modified functional responses are proposed for middle and top predators, and the Berec’s encounter-driven approach is used for cooperative hunting in the top predator. Various mathematical properties are investigated related to the model such as boundedness and persistence, equilibrium analysis, local and global stabilities of equilibrium points and the occurrence of various bifurcations. Extensive numerical simulation is performed for the better visualization of the impact of fear effect in both middle predator and prey, and the cooperative hunting among top predator population. Findings The results indicate that fear effect in both middle predator and prey populations can lead from stability to chaos. Furthermore, a high level of fear in prey and a moderate level of hunting cooperation among top predator is beneficial for the species’ coexistence. It is also obtained that the wasting time by middle predator due to fear of top predator promotes species’ coexistence, when the level of reduced encounter rate between top and middle predators is low or moderate. However, a high level of reduced encounter rate may be responsible for the extinction of top predator population. Originality/value This research proposed a modified functional response by considering realistic assumptions and provides the combined role of fear effect in prey and hunting cooperation among top predators, and the impact of reduced encounter rate between top and middle predators along with the wasting time by middle predator while hunting prey, due to antipredator behavior.