On the Inconsistency of Estimation with Uncertain Models

We study the hypothesis testing problem where an agent seeks to select a hypothesis from a finite set <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Theta$</tex>, based on a sequence of i.i.d. observations of a random variable <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$X_{k}\sim P^{\ast}$</tex> for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$k\geq 1$</tex> with unknown distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P^{\ast}$</tex>. Each hypothesis <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\theta^{\ast}\in\Theta$</tex> states that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$X_{k}\sim P_{\theta}$</tex>. The objective is to find a hypothesis <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\theta^{\ast}$</tex> such that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\theta= \arg\min\nolimits_{\theta\in\Theta}D_{KL}(P^{\ast}\Vert P_{\theta})$</tex>. However, the set of hypotheses <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\{P_{\theta}\}$</tex> is partially known, where only a finite number of observations are available for the random variables <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Y_{s}^{\theta}\sim P_{\theta}$</tex> with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\theta\in \Theta$</tex>. We show that contrary to classical Bayesian approaches, the obtained estimator will not be consistent, and the aggregated log-likelihood ratios will converge in distribution to a Gaussian distribution even when <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$k\rightarrow\infty$</tex>. Our result states that estimators with uncertain likelihoods will not concentrate on the true hypothesis. There is a strictly positive probability that the belief in a suboptimal hypothesis is maximal.