Convergence of the EM algorithm in KL distance for overspecified Gaussian mixtures

Abstract We present a study of the convergence properties of the Expectation-Maximization (EM) algorithm when applied to an overspecified model. In particular, we consider fitting a balanced mixture of two Gaussians to data originating from a single Gaussian. We provide theoretical bounds on the Kullback–Leibler (KL) divergence between the fitted and true distributions. An important feature is concavity and radiality of the expected log-likelihood function on a hypersurface induced by the EM algorithm, which greatly simplifies the analysis. We also show how our result on KL divergence can be used to upperbound the error rate of a mixture discriminant analysis classifier trained by the EM algorithm.