Out-of-Distribution Detection Using Maximum Entropy Coding and Generative Networks

Given a default distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P$</tex> and a set of test data <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x^{M}=\{x_{1},\ x_{2},\ \ldots,\ x_{M}\}$</tex> this paper seeks to answer the question if it was likely that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x^{M}$</tex> was generated by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P$</tex>. For discrete distributions, the definitive answer is in principle given by Kolmogorov-Martin-Lof randomness. In this paper we seek to generalize this to continuous distributions. We consider a set of statistics <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$T_{1}(x^{M}), T_{2}(x^{M}),\cdots$</tex>. To each statistic we associate its maximum entropy distribution and with this a universal source coder. The maximum entropy distributions are subsequently combined to give a total codelength, which is compared with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$-\log P(x^{M})$</tex>. We show that this approach satisfied a number of theoretical properties. For real world data <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P$</tex> usually is unknown. We transform data into a standard distribution in the latent space using a bidirectional generate network and use maximum entropy coding there. We compare the resulting method to other methods that also used generative neural networks to detect anomalies. In most cases, our results show better performance.