State space model for new generation network alternative to transformers: a survey

Abstract In the post-deep learning era, the Transformer architecture has demonstrated its powerful performance in pre-trained large models and various downstream tasks. However, the enormous computational demands of Transformers have deterred many researchers. To reduce the complexity of attention models, numerous efforts have been made to design more efficient methods. Among them, the state space model (SSM), as a possible replacement for the self-attention based Transformer model, has drawn more and more attention in recent years. This paper provides the first comprehensive review of SSM works and offers experimental comparisons and analyses to better demonstrate SSM’s features and advantages. First, we provide a detailed description of the principles to help the readers quickly grasp the key ideas of the SSM. Then, we will review existing SSMs and their various applications, including those in the fields of natural language processing, computer vision, graphs, multi-modal and multi-media, point cloud and event stream, and other domains. Next, we provide statistical comparisons and analyses of these models to help readers understand how different structures affect performance on various tasks. Finally, we suggest potential research directions that could further advance the development of the theoretical model and propose possible research points in this direction to better promote the application of SSMs.