Can we do beyond Wilks theorem for significance calculation? Estimating p-values with importance sampling

P-value estimation is particularly relevant in High Energy Physics for assessing the significance of potential discoveries. The reliance on Wilks' theorem, often used for p-value calculations, can be problematic for very small p-values where the asymptotic conditions may not hold. On the other hand, standard Monte Carlo simulations require prohibitively large sample sizes (~108). Importance sampling (IS) offers an alternative by focusing computational effort on the tails of the test statistic distribution. The core idea of IS involves sampling from a more convenient probability density function (pdf) and applying weights to ensure that the estimated expectations converge to the true values. In this work the problem is addressed on how to choose such pdf to permit a MC estimation with attainable sample sizes. It is shown that for models with a single parameter of interest (POI) exhibiting a monotonic relationship with the test statistic, importance sampling from the signal-plus-background (S+B) model’s pdf, with the parameter value set to its maximum likelihood estimate (MLE), significantly reduces the required number of pseudo-experiments to achieve a given precision, potentially by several orders of magnitude for 5 significance. To address more complex scenarios, such as two-sided hypotheses or models with multiple POIs, a method based on mixtures of pdfs corresponding to different parameter values that yield similar test statistic values is proposed. While less efficient and requiring a scan of the parameter space, this approach provides a path to estimate global p-values and move beyond the limitations of local p-value approximation. In summary, importance sampling presents a promising approach for calculating very small p-values when asymptotic methods are unreliable and direct Monte Carlo simulations are impractical. By strategically sampling from appropriately chosen pdfs and reweighting the events, this technique provides a more accurate and computationally feasible means to assess the significance of new physics signals. The methods discussed offer solutions for both simple and more complex models, paving the way for more robust statistical inference in high-energy physics analyses.