Statistic threshold of distinguishing the environmental effects and modified theory of gravity with multiple massive black-hole binaries

Abstract In future space-borne gravitational wave (GW) observations, matter around the sources might influence the evolution and GW signals from binary black hole (BBH) inspirals, which can be mistaken as deviations from general relativity (GR). Former research Phys. Rev. D 111 (2025) 104050 proposed a statistic F that characterizes the dispersion of measured parameters to distinguish environmental effect (dynamical friction (DF) from dark matter (DM) spike) and theory of modified gravity effect (varying G ). In this work we use the statistic to distinguish other pairs of effects with GW corrections at -4 PN order: DF from DM spike and the extra dimension theory, additionally determine a proper distinguishing threshold via Receiver Operating Characteristic Curve (ROC) curve method to avoid arbitrariness, especially when the two effects to compare have more overlap in the F distribution. Sources of different astronomical models are also considered, and two effects are still distinguishable but less than the example in former work Phys. Rev. D 111 (2025) 104050, so the threshold should be carefully selected. Following these procedures, we finally obtain the statistic thresholds of distinguishment between the three effects with GW corrections at -4 PN order: DF from DM spike, the extra dimension theory, and varying G theory. So each effect corresponds to its own interval of the statistic F , then future observation of F can pick the preferred effect. The method can be used to distinguish other effects among environmental effects and modified theories of gravity effects with the detection of GW events.