Optimal Decision Rules for Composite Binary Hypothesis Testing under Neyman-Pearson Framework

The composite binary hypothesis testing problem within the Neyman-Pearson framework is considered. The goal is to maximize the expectation of a nonlinear function of the detection probability, integrated with respect to a given probability measure, subject to a false-alarm constraint. It is shown that each power function can be realized by a generalized Bayes rule that maximizes an integrated rejection probability with respect to a finite signed measure. For a simple null hypothesis and a composite alternative, optimal single-threshold decision rules based on an appropriately weighted likelihood ratio are derived. The analysis is extended to composite null hypotheses, including both average and worst-case false-alarm constraints, resulting in modified optimal threshold rules. Special cases involving exponential family distributions and numerical examples are provided to illustrate the theoretical results.