Dual Soft-Adaptive Aggregation for Long-Context LLMs_ Mitigating Sequence-and-Depth Information Dilution

Long‑context large language models suffer from two coupled information‑dilution issues: flattened softmax weights and compressed KV representations along the sequence dimension, as well as progressively obscured early‑layer signals across the depth dimension. Existing approaches rely on hard token dropping or static layer‑wise block partitioning, which either permanently discard context information or fail to adapt to varying input complexity. This work presents a unified soft‑adaptive aggregation framework that formulates sequence‑level and depth‑level information dilution as a dual optimization problem. On the sequence side, we introduce Soft‑ChunkAttn, which divides historical context into semantic chunks via differentiable Gumbel‑Sigmoid boundary prediction. It adopts attention‑weighted chunk‑level summarization and conducts full‑scale softmax aggregation over all semantic chunks without hard top‑k pruning. An ALiBi‑style relative position mechanism is tailored for chunk summaries to enhance length extrapolation performance. On the depth side, we propose Virtual Dynamic Block‑AttnRes, which produces input‑aware virtual block summaries through inter‑layer‑similarity‑driven merging weights. A dedicated regularization term is introduced to stabilize training and mitigate granularity collapse. This is Version 2 of the preprint. This revision establishes the train‑inference closed‑loop for autoregressive KV‑cache streaming deployment. We put forward DS‑MLA‑Chunking (Delayed‑Split M‑Look‑Ahead Chunking), an autoregressive inference paradigm with delayed dynamic chunking, enabling Soft‑ChunkAttn to function under incremental‑decoding constraints. For Virtual Dynamic Block‑AttnRes, two inference modes are defined: a deterministic soft‑commit primary mode that minimizes the train‑inference discrepancy, and a threshold‑based hard‑commit alternative optimized for high‑throughput inference. We additionally add thorough complexity analysis, practical deployment considerations, and extended ablation protocols to assess the method’s effectiveness and deployability. Importantly, the underlying Transformer computation graph remains untouched. Vectorized segment‑wise operators are utilized for efficient GPU execution while maintaining compatibility with mainstream distributed training pipelines. This manuscript elaborates the full methodological design and theoretical analysis. Comprehensive experimental results including ablation metrics and large‑scale pre‑training evaluations will be provided in future revisions. Contact for discussion and collaboration: m3e1‑[email protected] This is a single‑authored preprint. The manuscript may contain imperfections and open issues. Constructive feedback, critical comments and potential research collaborations are welcome.