Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Authors
- Chong Ruan
- Damai Dai
- Lean Wang
- Wangding Zeng
- Yuqing Wang (ORCID: https://orcid.org/0009-0006-3161-0364)
- Liang Zhao (ORCID: https://orcid.org/0000-0002-5522-8630)
- Ming Zhang (ORCID: https://orcid.org/0009-0000-2631-1622)
- Junyu Luo (ORCID: https://orcid.org/0009-0001-6894-1144)
- Zhiping Xiao (ORCID: https://orcid.org/0000-0002-3165-6163)
- Huazuo Gao
- Yuxing Wei (ORCID: https://orcid.org/0009-0005-2364-6693)
- Wenfeng Liang
- Zhengyan Zhang
- Zhenda Xie
- Jingyang Yuan
Publication Details
- Published
- 2025-01-01
- DOI
- https://doi.org/10.18653/v1/2025.acl-long.1126
- Citations
- 32
- Primary Topic
- Stochastic Gradient Optimization Techniques
- Type
- article
- Field-Weighted Citation Impact
- 79.39
Funders
- National Natural Science Foundation of China
- Peking University
- National Key Research and Development Program of China