Estimating Intrinsic Dimension to Architect Efficient Neural Networks

Neural networks are pervasive in many industries due to their ability to dissect large multi-feature datasets. Because of this, it is important to implement parameter reduction to increase performance and prevent resource waste. Another motivation for reduction is to reduce attack vectors for adversaries, as well as features that may introduce unfair bias in a fully trained model. Intrinsic Dimensionality (ID) is a metric for the minimal number of features that adequately represent a given dataset. Accurately estimating this value may allow us to keep the most important features in the original data while reducing wasted features and potential security risks. Here, we investigate a set of local and global estimators as they operate on a series of datasets. This information is used to modify feedforward classifiers, and the efficiency of models with reduced latent spaces is measured. From our experimentation, IDs obtained using lPCA and Method of Moments (MoM) closely represent the true intrinsic dimension of linear and nonlinear datasets respectively; however, modifying latent spaces with these values results in poorer model performance. We also apply this data to a traffic sign dataset to demonstrate applications of ID estimation on data poisoning attacks, which suggests that lower latent spaces closer to ID estimations do not necessarily result in more robust networks.