Building a Dialogue Model using Generative Pre-trained Transformer

The pursuit of creating machines capable of engaging in natural and coherent conversation has been a longstanding focus of Embodied Conversational Agents (ECA) research. Initial endeavours to develop embodied conversational agents frequently employed rule-based methodologies, relying on pre-established templates and decision trees. Although these systems were effective in specific areas, they did not possess the versatility and adjustability necessary for unrestricted discussions. The emergence of machine learning, specifically the advancement of transformer-based models, has completely revolutionised the generative dialogue model for human conversation. Generally, there exist two primary approaches for modelling human conversation: retrieval-based and generative-based. Within retrieval-based systems, the response is constrained but more manageable to execute and susceptible to mistakes. In contrast, the generative-based model offers an unlimited response since it is generated. However, it demands significant resources, including training data and training computing power, to effectively model the data. This research makes use of the Daily Dialog dataset, which encompasses a diverse range of personalities, emotions, and dialogue scenarios. There are eight combinations of hyper-parameter tuning (pre-trained weights: Microsoft/DialoGPT-Medium and Microsoft/DialoGPT-Small, train batch size and the optimiser). In terms of the models’ performances, the Medium model generally exhibits a slightly higher performance in terms of F1 Score when compared to the Small model, with the highest F1 Score of 0.8 being achieved using the Adamw Torch optimiser with a training batch size of 16