Interference-Aware Concept Regularization: Scaling Circuit-Aware Low-Rank Adaptation with Disentangled Feature Constraints

We introduce Interference-Aware Concept Regularization (IACR), a training objective that replaces L1 sparsity penalties with pairwise interference constraints during Circuit-Aware Low-Rank Adaptation (LoRA) fine-tuning. Drawing on the mechanistic understanding of interference weights introduced by Anthropic (2025), we compute pairwise cosine similarities between concept projections through the query and key attention circuits at every transformer layer. The resulting interference regularization term penalizes the adapter for introducing features that would interfere with the base model's pre-existing circuit structure, preserving mechanistic integrity while adapting to new domains. Applied to three security LLMs—Averroes, SAIF, and Rushd—IACR reduces concept interference by 34% while maintaining task accuracy, improves feature disentanglement in J-lens circuit mapping by 28%, and enables the use of higher LoRA ranks without inducing catastrophic forgetting. Our results demonstrate that interference-aware constraints offer a principled alternative to L1 sparsity for preserving model circuits during fine-tuning.