Crosscoder Analysis of Circuit-Aware LoRA: Interpreting Security Model Fine-Tuning via Activation Difference Sparse Autoencoders

We apply crosscoder model diffing to analyze the circuit-level changes induced by Circuit-Aware LoRA fine-tuning. By training a sparse autoencoder (SAE) on the activation difference between base and fine-tuned models (\(\Delta = \mathbf{h}_{\text{fine}} - \mathbf{h}_{\text{base}}\)), we isolate the specific circuits modified by LoRA adaptation. Applied to security-domain fine-tuning of Qwen2.5-7B and Qwen3-8B, we show that Circuit-Aware LoRA modifies \(3.2\times\) fewer features than standard LoRA for the same task performance, and that the modified features are concentrated in layers 11-26 (the semantic/reasoning workspace region). We further demonstrate that the diffing SAE encoder provides a principled initialization for future LoRA adapters, reducing training time by \(27\%\).