QK-Guided Circuit-Aware LoRA: Structured Low-Rank Adaptation via Attention Circuit Factorization for Security Language Models

Abstract. We extend Circuit-Aware LoRA by introducing QK-guided rank selection, a principled method for allocating LoRA rank across transformer layers based on the intrinsic dimensionality of attention QK circuits. Inspired by Anthropic's finding that attention QK circuits form low-rank bilinear forms interpretable as feature detectors, we apply Singular Value Decomposition (SVD) to the product \(W_Q^\top W_K\) at each layer to determine per-layer LoRA rank allocation. This preserves attention circuit structure while minimizing adapter parameters. Applied to Qwen2.5-7B (security QA, 28 layers) and Qwen3-8B (multi-expert reasoning, 36 layers) across three security-focused fine-tuning suites (Averroes, SAIF, Rushd), we show that QK-guided rank selection reduces total adapter parameters by 40% while maintaining or improving task performance compared to uniform rank allocation, and achieves within 1% of full fine-tuning performance on security-domain benchmarks.