When the Expected Jacobian Misleads: A Validity-Domain Map for E[J]-Lens Measurements

A Jacobian lens summarises a language model's local input–output map by one matrix per layer: the expectation J̄_ℓ = E[∂h_final/∂h_ℓ] of the per-input Jacobian over a fitting corpus. The originating report is explicit that this averaging trades exactness on any one prompt for a fixed, context-independent map. We measure the size of that trade — the self-consistency of the averaged map with the per-input Jacobian it summarises — on 14 checkpoints from four model families. On 11 of them the transported vector J̄_ℓ h agrees in direction with the per-input Jacobian–vector product, with pooled median cosine from +0.0745 to +0.9534. On three it does not merely degrade but reverses sign: the Gemma-3 base checkpoints at 4B, 12B and 27B give −0.127, −0.475 and −0.5104, while the same architecture at 1B is untouched at +0.903. A version comparison under matched activation statistics (instruction-tuned/base activation-maximum ratios of 1.01 and 1.04) removes the sign flip entirely (−0.127 → +0.745, −0.475 → +0.853), dissociating it from activation magnitude, model size, base training in general, and model family as the driver. Two out-of-sample predictions fixed before the 27B measurement were both correct. Where the trade fails it fails in kind, not in degree, and whether it fails is a property of the (architecture × pretraining-state) pairing rather than of the method.