Value Residual Learning

While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge.Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers.This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals.And a variant is SVFormer, where all layers share the first layer's value embedding.Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11% fewer model parameters and 20.3% less training data compared to Transformer, while maintaining similar memory usage and computational cost.Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KVefficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.