A mathematical perspective on transformers
Transformers play a central role in the inner workings of large language models.We develop a mathematical framework for analyzing transformers based on their interpretation as interacting particle systems, with a particular emphasis on long-time clustering behavior.Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.
Authors
- Philippe Rigollet (ORCID: https://orcid.org/0000-0002-1554-8912)
- Cyril Letrouit (ORCID: https://orcid.org/0000-0001-5469-9249)
- Yury Polyanskiy
- Borjan Geshkovski (ORCID: https://orcid.org/0000-0002-7890-3352)
Institutions
- Centre National de la Recherche Scientifique (FR)
- Université Paris-Saclay (FR)
- Laboratoire de Mathématiques d'Orsay (FR)
- Massachusetts Institute of Technology (US)
Publication Details
- Journal
- Bulletin of the American Mathematical Society
- Published
- 2025-04-29
- DOI
- https://doi.org/10.1090/bull/1863
- Citations
- 9
- Primary Topic
- Induction Heating and Inverter Technology
- Type
- article
- Field-Weighted Citation Impact
- 6.42
Funders
- National Science Foundation
- Agence Nationale de la Recherche
- CHIST-ERA