Topic counts and quality in topic models for historic corpora

Abstract Topic modelling methods enable the identification of potential topics within a corpus of historical texts; in particular, they enable the identification of latent topics that are not described just by a single word. Like so many computational methods for the automatic processing of historical text corpora, they come with a number of parameters with which the method can be tuned and adapted. Each change in the settings of any of these parameters will generate a new set of topics that will differ in larger or smaller ways and which may be qualitatively better or worse. One of the main parameters for tuning topic models is setting the number of topics to be generated. In this article, we present an analysis of the impact of the number of topics on the quality of topic models for two historical text corpora. Two manual evaluation approaches are combined with an automated evaluation metric, and based on the results, we propose a formalized process for choosing the final set of parameters for a topic model. The process ensures the quality of the final model, while minimizing the amount of manual evaluation work. The more structured process also allows for better documentation of the parameter choices and in that way enables better replicability of any research using topic models.