Statistical properties of mean relative entropy and its applications

A fundamental concept in information theory is information entropy, which is used to quantify the degree of uncertainty associated with random variables. Building upon this, relative entropy serves as a measure of the discrepancy between two probability distributions and has been widely studied in statistics. The function of relative entropy in the field of parameter estimation has been extensively investigated. This paper extends existing research by deriving minimum mean relative entropy estimators for the parameters of the Gamma, Laplace, and Rayleigh distributions. Furthermore, we introduce the residual mean relative entropy, a novel measure based on mean relative entropy, and apply it to model comparison. To estimate this measure, we employ kernel density estimation and bootstrap methods. Simulation experiments and empirical analysis demonstrate that the proposed residual mean relative entropy provides an effective new criterion for model comparison.