Activation Sparsity Does Not Independently Improve Neuron Monosemanticity: An Apparent Effect Explained Away by Network Depth

Activation sparsity is widely used to accelerate large language model inference, and is often claimed to make neurons more interpretable, or monosemantic. We test this claim directly, comparing an efficiency-driven ReLU-sparsified model (MiniCPM-S-1B) against its dense SiLU sibling on per-layer neuron monosemanticity, measured by both a Jensen–Shannon concept-separability score and a variance-based η² score. While the sparse model appears more monosemantic in deeper layers, we show this advantage is fully explained by network depth: after regressing out depth, per-layer sparsity has no detectable correlation with the monosemanticity gain (residual r ≈ 0.12). This null result is robust across two orthogonal concept dimensions (topic and sentiment) and two methodologically distinct metrics. Our finding stands in tension with reports of sparsity-induced monosemanticity in mixture-of-experts models, suggesting that any link between sparsity and monosemanticity may depend on the specific form of sparsity rather than holding universally.