Extended-Attention Applied in Selected Stages

The Self-attention mechanism was widely applied in Computer Vision (CV) field. However, while Self-attention mechanism brings high accuracy to Transformer, it also brings a huge amount of calculation. To further enhance accuracy and decrease computational load, in this paper, two refinements were implemented. Firstly, two matrices were introduced to enhance the extraction of image information for Extended-attention method. The capacity of Transformer was enhanced by these matrices to capture crucial information in complex tasks, the final accuracy and generalization performance were significantly improved. Secondly, the impact of implementing the aforementioned Self-attention mechanism at various stages of the Transformer is investigated. Intriguingly, our findings reveal that selectively implementing this method at specific stages leads to a significant enhancement in the model's performance.