Black Hole Binary Mergers in Galactic Nuclei: Combined Effects of Secular Dynamics and Stellar Flybys

Abstract Galactic nuclei, with their central supermassive black holes (SMBHs) and surrounding dense star clusters, serve as natural laboratories for studying gravitational dynamics in extreme gravitational environments. In this work, we model the long-term evolution of black hole binaries (BHBs) in these environments by incorporating the tidal influence of the central SMBH, the potential of the nuclear star cluster, SMBH-induced relativistic precession, and stochastic perturbations from stellar flybys. Our numerical simulations show that the long-term evolution of BHBs is predominantly governed by the SMBH-driven Zeipel–Lidov–Kozai (ZLK) mechanism. However, once stellar flybys are included, they can modify both the binary angular momentum and energy. The cumulative effect of repeated flyby encounters substantially reshapes the long-term dynamical evolution, eventually leading systems toward either merger or disruption. Based on a population synthesis study, we find that this combined secular and nonsecular channel produces a merger rate of approximately R ≃ 16 Gpc − 3 yr − 1 , which is significantly higher than the rates predicted by dynamical channels that neglect stellar flybys. Furthermore, we predict that a nonnegligible fraction of merging systems would retain detectable eccentricities at low frequencies around 5 Hz, thus providing a distinctive observational signature for next-generation ground-based gravitational-wave (GW) detectors. Our results demonstrate that the stochastic perturbations from the surrounding environment are essential for modeling BHB evolution in galactic nuclei, and that this channel may contribute a major fraction of observed mergers.