A statistical journey to DNN, the third trip: Language model and transformer

Over the past decade, the remarkable advancements in deep neural networks have paralleled the development and evolution of language models.Initially, language models were developed in the form of Encoder-Decoder models using early RNNs.However, with the introduction of Attention in 2015 and the emergence of the Transformer in 2017, the field saw revolutionary growth.This study briefly reviews the development process of language models and examines in detail the working mechanism and technical elements of the Transformer.Additionally, it explores statistical models and methodologies related to language models and the Transformer.