Nonlinear dynamics in a fear‐driven predator–prey system: Bistability, bifurcations, hydra effect, and optimal harvesting

The impact of predator‐driven fear on ecosystems is significant and can encompass both trophic (direct) and nontrophic (indirect) effects. Previous studies have shown that nontrophic fear effects have an important role in predator–prey dynamics. This study investigates the nontrophic fear effect on prey caused by generalist predators and explores optimal harvesting. We assume that the reproduction rate of prey is reduced due to the fear effect, and generalist predator follows Holling type II foraging strategy for predation. Additionally, we assume that predators are commercially valuable and harvested proportionately to their density. We demonstrate the existence of equilibrium points, their local and global stability, and bifurcation analysis. We observe that the model system undergoes a sequence of codimension one and codimension two bifurcations. Our results show that in the absence of predator harvesting, increasing levels of fear destabilize the predator–prey system, and controlled harvesting is beneficial for the coexistence of both populations. Also, the harvesting of predators may produce hydra and multiple hydra effects. We identify different types of bistability phenomena, which emphasize the importance of initial population size. Further, an optimal harvesting policy is also explored by formulating an optimal control problem (OCP). The harvesting cost‐functional is designed by incorporating the bionomic equilibrium state. We use Pontryagin's maximum principle and solve the OCP numerically. It is observed that implementing optimal harvesting not only contributes to the ecological benefits by maintaining a sustainable balance of predator–prey evolution but also plays a significant role in maximizing the economic benefits.