On frequentist confidence intervals in a non-Gaussian regime

Abstract We study frequentist confidence intervals based on graphical profile likelihoods (Wilks’ theorem, likelihood integration), and the Feldman–Cousins (FC) prescription, a generalisation of the Neyman belt construction, in a setting with non-Gaussian Markov chain Monte Carlo (MCMC) posteriors. Our simplified setting allows us to recycle the MCMC chain as an input in all methods, including mock simulations underlying the FC approach. We find all methods agree to within $$10 \%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>10</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> in the close to Gaussian regime, but extending methods beyond their regime of validity leads to greater discrepancies. As a key consistency check, we recover a shift in cosmological parameters between low and high redshift cosmic chronometer data with the FC method, but only when one fits all parameters back to the mocks. We observe that fixing parameters, a common approach in the literature, risks underestimating confidence intervals.