Flavor phenomenology of an extended 2HDM with the inverse seesaw mechanism

Abstract We perform a detailed and comprehensive study of several flavor physics observables in both the lepton and quark sectors within the framework of an extended two-Higgs-doublet model theory where the inverse seesaw mechanism is implemented to generate the Standard Model (SM) fermion mass hierarchy. In that theory, the SM gauge symmetry is supplemented by the local $U(1)_X$ and discrete $Z_4\times Z_2$ groups. In particular, we find that the leptonic flavor observables, specifically the branching ratios of charged lepton flavor-violating decays $\mu \rightarrow e\gamma , \tau \rightarrow e(\mu )\gamma$ as well as the anomalous magnetic moments $\Delta a_{e(\mu )}$, strongly depend on the couplings of the neutral CP-even (-odd) Higgses with an exotic charged lepton $E_1$, whereas other observables involving three-body leptonic decays BR($l\rightarrow 3l^{\prime }$), Mu–$\overline{\text{Mu}}$ transition, and coherent conversion $\mu \rightarrow e$ in a muonic atom are predicted to be less than several orders of magnitude compared to the corresponding experimental limits. Regarding the quark flavor observables, the most stringent limits arising from the flavor-changing neutral currents (FCNCs) are those involving the down-type quark $d_a\rightarrow d_b$ ($a=1,2,3$) transitions and including the branching ratios of inclusive decay BR($\bar{B}\rightarrow X_s \gamma$), pure leptonic decay of the $B_s$ meson BR$(B_s\rightarrow \mu ^+\mu ^-)$, and meson mixing $ \Delta m_{K,B_s, B_d}$. Considering the constraints obtained from these observables, the new physics contributions to other processes such as BR$(B_s\rightarrow \tau ^+\mu ^-)$, BR$(B^+\rightarrow K^+\tau ^+\tau ^-)$, and BR$(B^+\rightarrow K^+\tau ^+\mu ^-)$, as well as the flavor-changing charged current $b\rightarrow c$ transition, specifically lepton flavor universality-violating ratios $R_{D^{(*)}}$, are shown to be remarkably small. Regarding the observables in the up-type quark transitions, the FCNC top quark processes $t\rightarrow u(c)\gamma$ and $t\rightarrow u(c)h$ have branching ratios consistent with the experimental limits. Additionally, observables related to SM-like Higgs boson decays, such as the lepton flavor-violating decays BR$(h\rightarrow \bar{l}^{\prime }l)$ and modified couplings $a_{h\bar{f}f}$, are also discussed.