Permutation-Invariant Transformers for Attribute Embeddings in Information Retrieval

Generating robust embeddings for unordered sets of phrases is a challenging problem in NLP, particularly when embeddings must exhibit permutation invariance at the set level while preserving the inherent order of tokens within each phrase. This challenge is especially relevant in e-commerce, where product attributes such as size, color, and dimensions are often unstructured and vary across products. Embedding these unordered attribute sets is critical for query-product understanding, search ranking, and recommenda- tion systems. Existing set embedding methods have primarily focused on domains such as graphs, with limited applicability to NLP tasks. To address this gap, we propose a Phrase-Localized Attention Network (PLAN), a Transformer-based model that ensures intra-attribute order preservation while enabling permutation-invariant representations at the set level. Experimental results on the Amazon Shopping Queries dataset demonstrate that PLAN outperforms existing models, highlighting its effectiveness in search applications.