Exploring the Semantic Representations of Text in Subspaces of Latent Space: A Case Study on Color

Language models like BERT have advanced the representation of textual semantics in high-dimensional latent spaces, enabling numerous natural language processing applications. However, their capacity to represent domain-specific concepts, such as "color," remains underexplored. This study investigated the semantic representation of text in color concept subspace of latent space. Using embeddings of nearly 1,000 color names from the XKCD color survey generated by BERT, we identified limitations in BERT’s ability to cluster perceptually similar colors. To address this, we proposed a supervised learning approach to project embeddings into a color-specific subspace, isolating and enhancing color semantics. Experimental results demonstrated the methodology’s effectiveness in improving semantic clustering through qualitative and quantitative evaluations. Moreover, our general approach not only explored the concept of color but also provided the possibility of exploring and disentangling semantic subspaces for other domain-specific concepts, contributing to the understanding and manipulation of latent space structures in language models.