Directional Complexity of SIGReg with Anisotropic Gaussian Targets

Sketched Isotropic Gaussian Regularization (SIGReg) aligns learned representations with an isotropic Gaussian target along random one-dimensional projections. LeJEPA establishes finite-direction guarantees because the rotational symmetry of this target makes every direction identical. Structured representations, however, require anisotropic targets that vary across directions. We prove that while target anisotropy preserves the geometric approximation rate, directional complexity depends on the target’s centered spectral moments and spectral endpoints. We show empirically that across targets with matched condition numbers, spectral shape governs direction budgets. These budgets scale with the target’s spherical Dirichlet energy and initial slicing-gradient variance, while its leading angular harmonic couples directly to directional sampling error. To eliminate this quadratic coupling, we introduce Orthogonal Directional Slicing (ODS), which draws complete random orthonormal frames that enforce exact second-moment balance. While ODS reduces gradient variance across all conditions, it does not reliably lower direction budgets, as higher-degree angular harmonics dominate directional complexity. Reducing this complexity therefore requires canceling these higher-order harmonics, rather than simply drawing more directions.