Forecast analysis of astrophysical stochastic gravitational wave background beyond general relativity: a case study on brans-dicke gravity

Abstract Scalar-tensor gravity, exemplified by Brans-Dicke (BD) gravity, introduces additional scalar polarization modes that contribute scalar radiation alongside tensor modes. We conduct a comprehensive analysis of how gravitational wave generation and propagation effects under Brans-Dicke gravity are encoded into the astrophysical stochastic gravitational wave background (AGWB). We perform end-to-end analyses of realistic populations of simulated coalescing binary systems to generate AGWB mock data with third-generation gravitational wave detectors and conducted a complete Bayesian analysis for the first time. We find the uncertainties in the population properties of binary black holes (BBH) significantly affect the ability to constrain BD gravity. Furthermore, we explore the detectability of potential scalar backgrounds that originates from binary neutron star (BNS) and neutron-star-black-hole (NSBH) mergers, with NSBH systems expected to modify the spectral index of the scalar background and introduce oscillatory behavior. We show that the observations of the AGWB enable the separation of mixed tensor and scalar polarization modes with comparable sensitivity to each mode. However, the scalar background is expected to remain substantially weaker than the tensor background, even in scenarios where BD gravity exhibits significant deviations from general relativity (GR), resulting only upper limits can be placed on the scalar background. We conclude that for ambiguous populations, employing waveform matching with individual sources provides a more robust approach to constrain BD gravity.