GA: Using Global Memory Tokens to Fuse Global Features

The traditional multi-head self-attention mechanism models the relationships between elements within a sequence through the interaction between queries, keys, and values, but has certain limitations in capturing global information. To address this issue, this paper proposes an improved self-attention mechanism - global self-attention mechanism, aimed at enhancing the global information capture capability and robustness of Transformer models in sequence modeling tasks. This mechanism introduces a global context vector as the fourth input variable based on the classic QKV structure, endowing the model with the ability to perceive the entire sequence state. Specifically, each query can not only focus on local information in the sequence, but also simultaneously consider the global context, thereby better capturing global dependencies. The innovation of this method lies in the introduction of a global memory token, which enables the model to integrate global information and enhance its sensitivity to long-term trends and global patterns. In summary, this method not only has strong practical value, but also is easy to implement and has a wide range of application prospects, suitable for various sequence modeling tasks.