Dimensionality Reduction for Embeddings: A Pattern-Based Approach with Comparative Benchmarks

This article suggests a new and innovative method of embedding dimensionality reduction using variance-based selection methods to achieve maximum storage with optimal accuracy. By leveraging the central limit theorem, we identify specific dimensions with significantly low variance when the embedding engine processes a limited range of semantic meanings. Using diverse datasets, we demonstrate that a small trade-off in accuracy can yield substantial memory savings by mapping and omitting dimensions with minimal impact. This technique is particularly beneficial when embeddings are stored in data repositories. Experiments are conducted across various embedding models to validate the robustness of the method. Additionally, the approach is compared with OpenAI's dimensionality reduction feature in their embedding-large-3 model, highlighting the respective advantages and limitations of each method.