Text Prediction Using Attention Mechanism

This research paper will look at how attention mechanisms can work when applied to Transformer models, in the context of achieving better performance for predicting the text of movie scripts. Attention mechanisms have been responsible for major breakthroughs in Natural Language Processing (NLP) in the recent years due to its ability to focus on most relevant parts of an input sequence. Through the use of the Transformer architecture that was introduced in the ‘Attention is All You Need’ paper, this work explores how attention can model long-range context in text, leading to better predictions than vanilla models. We implement Transformer model from scratch in Python with multi-head attention, feed-forward layers, and softmax activation functions. Extensive experiments on movie script datasets rigorously evaluate the performance of our model compared with state-of-the-art next-word prediction models, showing a promise of much higher word prediction accuracy of our model in movie script contexts.