ConText Mining: Complementing Topic Models with Few-Shot In-Context Learning to Generate Interpretable Topics

Topic models are widely used across various domains to effectively cluster text-based documents and reveal their underlying themes. However, representing these clustered topics using keywords poses several challenges: difficulties in interpreting the topics, scalability issues in large web-based applications, and the potential for biased interpretations. To address these limitations, this study proposes complementing topic model outputs - the keywords representing clustered documents - with few-shot in-context learning (ICL) to generate coherent and interpretable topic labels. To assess the effectiveness of this approach, a labeled dataset of topics from Yelp reviews and New York Times (NYT) articles was utilized. Keywords generated from Yelp reviews, spanning five distinct topic modeling techniques - LDA, CorEx, ETM, BERTopic, and TopClus - along with their corresponding labels, were provided as few-shot examples to label topic modeling outputs of the NYT articles, and vice versa. Cosine similarities were calculated to measure the semantic similarities between the generated labels and the ground-truth labels. The results demonstrate that incorporating few-shot ICL can produce coherently labeled topics, achieving an average similarity of up to 80.9%. Overall, few-shot ICL outperformed existing prompt engineering methods in labeling topics from the topic modeling outputs by an average of 8.6%, even when few-shot examples were drawn from drastically different domains. However, performance was significantly correlated with the quality and coherence of the keywords generated by the topic models. Topic models that represented clusters with a less coherent set of keywords typically generated weaker, less interpretable, and semantically less accurate labels.