Efficient Embedding Initialization via Dominant Eigenvector Projections
The embedding layer is essential in deep learning, transforming high-dimensional data into compact representations. However, growing datasets and model sizes pose challenges in training time, memory, and generalization. We propose a scalable method for embedding initialization via spectral dimensionality reduction using dominant eigenvector projections.
Authors
- Quentin Petit (ORCID: https://orcid.org/0009-0006-8764-9081)
- Chong Li (ORCID: https://orcid.org/0000-0002-4160-7170)
- Jack Dongarra (ORCID: https://orcid.org/0000-0003-3247-1782)
- Nahid Emad (ORCID: https://orcid.org/0000-0002-4228-0069)
Institutions
- Université Paris Sciences et Lettres (FR)
- Université Paris-Saclay (FR)
- Centre de Recherche en Informatique (FR)
- Huawei Technologies (France) (FR)
- Maison de la Simulation (FR)
- Laboratoire d’informatique Parallélisme Réseaux Algorithmes Distribués (FR)
- University of Tennessee at Knoxville (US)
Publication Details
- Published
- 2025-11-07
- DOI
- https://doi.org/10.1145/3731599.3767541
- Citations
- 1
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- article
- Field-Weighted Citation Impact
- 2.48