Simultaneous Hierarchical Topic Selection and Initialization for Latent Nested Dirichlet Allocation

This paper proposes a unified framework that simultaneously addresses three key limitations of Latent Dirichlet Allocation (LDA): (1) inability to model topic correlations, (2) sensitivity to initialization, and (3) difficulty in selecting the optimal number of topics. Despite the widespread utilization of LDA in the context of topic modeling, there is a lack of research attention to the importance of such limitations. To this end, we introduce a hierarchical topic model based on the Nested Dirichlet distribution, called Latent Nested Dirichlet Allocation (LNDA). Our novel framework includes the derivation of the complete variational EM inference for LNDA with a novel Hessian factorization, and integrating pruning/splitting initialization with RPC-based topic selection within the hierarchical NDD structure. Therefore, it initializes the model while simultaneously selecting the optimal number of topics based on the perplexity and rate of perplexity change, respectively. This is achieved through iteratively pruning low-probability topics or splitting overly broad ones. The hierarchical nature allows topics to appear in a nested structure, which naturally provides the correlation information among them. Our novel framework shows a mean improvement in terms of perplexity, coherence, and classification accuracy by approximately 15–90%, 5–125%, and 20–170%, respectively, as derived from the experimental results across datasets.