Probing a nonminimal coupling through superhorizon instability and secondary gravitational waves

In this paper, we investigate the impact of scalar fluctuations ($\ensuremath{\chi}$) nonminimally coupled to gravity, $\ensuremath{\xi}{\ensuremath{\chi}}^{2}R$, as a potential source of secondary gravitational waves (SGWs). Our study reveals that when reheating EoS ${w}_{\ensuremath{\phi}}<1/3$ and $\ensuremath{\xi}\ensuremath{\lesssim}1/6$ or ${w}_{\ensuremath{\phi}}>1/3$ and $\ensuremath{\xi}\ensuremath{\gtrsim}1/6$, the superhorizon modes of scalar field experience a tachyonic instability during the reheating phase. Such instability causes a substantial growth in the scalar field amplitude leading to pronounced production of SGWs in the low and intermediate-frequency ranges that are strong enough to be detected by Planck and future gravitational wave detectors. Such growth in superhorizon modes of the scalar field and associated GW production may have a significant effect on the strength of the tensor fluctuation at the cosmic microwave background (CMB) scales (parametrized by $r$) and the number of relativistic degrees of freedom (parametrized by $\mathrm{\ensuremath{\Delta}}{N}_{\mathrm{eff}}$) at the time of CMB decoupling. To prevent such overproduction, the Planck constraints on tensor-to-scalar ratio $r\ensuremath{\le}0.036$ and $\mathrm{\ensuremath{\Delta}}{N}_{\mathrm{eff}}\ensuremath{\le}0.284$ yield a strong lower bound on $\ensuremath{\xi}$ for ${w}_{\ensuremath{\phi}}<1/3$, and upper bound on the value of $\ensuremath{\xi}$ for ${w}_{\ensuremath{\phi}}>1/3$. Taking into account all the observational constraints we found the value of $\ensuremath{\xi}$ should be $\ensuremath{\gtrsim}0.02$ for ${w}_{\ensuremath{\phi}}=0$, and $\ensuremath{\lesssim}4.0$ for ${w}_{\ensuremath{\phi}}\ensuremath{\ge}1/2$ for a wide range of reheating temperature within ${10}^{\ensuremath{-}2}\ensuremath{\lesssim}{T}_{\mathrm{re}}\ensuremath{\lesssim}{10}^{14}\text{ }\text{ }\mathrm{GeV}$, and for a wide range of inflationary energy scales. Further, as one approaches ${w}_{\ensuremath{\phi}}$ toward $1/3$, the value of $\ensuremath{\xi}$ remains unconstrained. Finally, we identify the parameter regions in the $({T}_{\mathrm{re}},\ensuremath{\xi})$ plane, which can be probed by the upcoming GW experiments namely BBO, DECIGO, LISA, and ET.