Bi‐Directional Recurrent Attentional Topic Model Using Flexible Priors

ABSTRACT This article presents extensions to the Bi‐Directional Recurrent Attentional Topic Model (bi‐RATM) framework, a Dirichlet‐based model used in text document analysis. The allocation of topics to a sentence in a document is determined by its content as well as the topics of its neighboring sentences, and the weighting is typically variable. Many current Bayesian approaches to document modeling do not consider these factors. The bi‐RATM model fills this gap in document embedding by implementing the attention mechanism to model both the relationships between consecutive sentences and the sequential order of sentences. The restrictions of the Dirichlet prior (independent assumption within its components and highly constrained covariance structure) tend to prevent efficient processing given the huge use of count data in modeling today. To address this issue, we propose alternative models with different priors, such as generalized Dirichlet (GD) and Beta‐Liouville (BL), which lead to GD‐bi‐RATM and BL‐bi‐RATM models, respectively. We also implement a deterministic method for fast convergence in the proposed algorithms that employ variational Bayesian inference. By assigning each observation to a distinct cluster, these two models provide higher flexibility. Using real‐world applications, we generate several multitopic models and evaluate their strengths and weaknesses. The performance of the models is evaluated based on their perplexity. Our results show that the two proposed models outperform other commonly used topic modeling models in terms of performance.