Order Inversions in Aggressive Low-Rank Compression: Activation Choice, Training Regime, and the Metric

Two decisions made before a network is compressed are usually treated as settled by uncompressed performance: which activation function to use, and whether to compress after training or to train under the rank constraint from the start. We show that both decisions reverse under aggressive low-rank truncation, and that the two reversals share a boundary. Across fifteen activation functions, five image datasets and five seeds, the ordering of activations by accuracy after truncation to sixteen singular directions is close to the reverse of their ordering before truncation, with rank correlations of -0.82 on Fashion-MNIST and -0.72 on CIFAR-10. Comparing networks trained under a rank constraint against dense networks truncated to the same storage cost, the constrained network wins in all 60 configurations on CIFAR-10, by 23.4 percentage points at rank eight, and its advantage correlates at 0.982 with the accuracy that truncation destroys, a relation that repeats on Fashion-MNIST at 0.993. The activations that survive truncation best are the ones that learn worst under the constraint. Both reversals disappear near rank sixty-four. Four candidate mechanisms for the size of the effect fail under controlled tests, a fifth fails a pre-registered causal test, and a sixth, decided by a pre-registered experiment on 270 cells, identifies the range of the activation's output as a determinant. On the same cells, the correlation between the squared derivative of the activation and the accuracy difference changes sign, from +0.51 to -0.84, when the difference is normalized by the uncompressed baseline, so a correlation reported without naming its metric is not interpretable. Every number in the paper is generated from the result files by one script, and a separate battery ties each of them back to those files. What follows for practice is short. A practitioner who intends to compress should decide first whether compression will be applied after training or during it, because that decision reverses the choice of activation. In the datasets where the reversal is strong, an activation selected by uncompressed accuracy is a worse choice under aggressive truncation than one selected at random, and above the boundary the uncompressed choice carries over. And a correlation between the derivative of an activation and the accuracy it preserves should not be reported without naming the metric, since the sign depends on it. Pre-print in progress. The file is restricted while the manuscript is under advisor review. Access is granted by share link.