Evaluating text embeddings for two-dimensional text corpora representations

Several text corpus visualizations utilize a map-like metaphor, where the layout reflects the semantic similarity between documents. The underlying two-dimensional scatterplots are created by combining a latent embedding with a subsequent dimensionality reduction. In this work, we analyze the impact of embedding quality on layout quality. We evaluate the accuracy of the layout, specifically the preservation of local and global structures of the text corpus in its two-dimensional representation. Additionally, we assess class separation, focusing on the effectiveness of distinguishing classes within the two-dimensional space. We introduce a benchmark <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mrow> <mml:mi mathvariant="script">B</mml:mi> <mml:mo>=</mml:mo> <mml:mrow> <mml:mo>(</mml:mo> <mml:mi mathvariant="script">D</mml:mi> <mml:mo>,</mml:mo> <mml:mi mathvariant="script">L</mml:mi> <mml:mo>,</mml:mo> <mml:msub> <mml:mrow> <mml:mi mathvariant="script">Q</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>E</mml:mi> </mml:mrow> </mml:msub> <mml:mo>,</mml:mo> <mml:msub> <mml:mrow> <mml:mi mathvariant="script">Q</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>DR</mml:mi> </mml:mrow> </mml:msub> <mml:mo>)</mml:mo> </mml:mrow> </mml:mrow> </mml:math> consisting of a collection of text corpora <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mrow> <mml:mi mathvariant="script">D</mml:mi> </mml:mrow> </mml:math> , a set of layout algorithms <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mrow> <mml:mi mathvariant="script">L</mml:mi> </mml:mrow> </mml:math> that combine text embeddings with dimensionality reductions, quality metrics <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mrow> <mml:msub> <mml:mrow> <mml:mi mathvariant="script">Q</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>E</mml:mi> </mml:mrow> </mml:msub> </mml:mrow> </mml:math> for evaluating text embeddings, and quality metrics <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mrow> <mml:msub> <mml:mrow> <mml:mi mathvariant="script">Q</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>DR</mml:mi> </mml:mrow> </mml:msub> </mml:mrow> </mml:math> for assessing accuracy and class separation. We generate a multivariate dataset by evaluating this benchmark, which we further analyze in a descriptive analysis. Our results indicate that, for Latent Semantic Indexing combined with tf-idf weighting and t-distributed Stochastic Neighbor Embedding, coherence plays a substantial role in determining the accuracy of the layout. Additionally, our findings reveal that embeddings do not enhance class separation in the two-dimensional scatterplot representation. As main result, we provide more fine-grained guidelines for effectively utilizing text embeddings and dimensionality reduction techniques to generate two-dimensional scatterplot representations of text corpora reflecting semantic similarity.