Confidence intervals using Turing's estimator: simulations and applications

Consider the problem of sequentially sampling categorical data from a finite or countably infinite alphabet. After observing n data points, we aim to estimate the rth occupancy probability, which is the total probability of all outcomes that appear exactly r times in the sample. The 0th occupancy probability, often called the missing mass, is the probability of observing a previously unseen outcome. One of the best-known estimators for these probabilities is Turing's estimator. In this paper, we study and compare several confidence intervals (CIs) for the occupancy probabilities that are based on Turing's estimator. We perform a simulation study to better understand the finite-sample performance of these CIs and introduce an approach for selecting the appropriate CI for a given sample. We also present an application to the problem of authorship attribution and apply it to a dataset consisting of tweets from users on X (formerly Twitter). Furthermore, we derive theoretical results on the asymptotic distribution of Turing's estimator for geometric and discrete uniform distributions.