Dissecting the Frog: Understanding ChatGPT's Sense of Humour

As the popularity surrounding large language models like OpenAI’s GPT series grows, it is crucial to explore areas where they still underperform, particularly in understanding humour. While models such as ChatGPT have demonstrated state-of-the-art results on benchmark tests (Brown et al., 2020), studies like Borji (2023) highlight their consistent failures in explaining jokes. This research investigates why such failures occur by analysing ChatGPT’s ability to interpret humour using the General Theory of Verbal Humour (GTVH) (Attardo & Raskin, 1991). GTVH suggests that jokes arise from the interplay of two semantically overlapping but fundamentally opposed scripts. This study hypothesises that ChatGPT’s architecture would be able to identify these opposed scripts. To test this, ChatGPT was asked to analyse 14 variants of two novel jokes, each crafted according to GTVH’s seven differences. Each version was presented five times, and the responses were qualitatively analysed for patterns in success and failure. The findings showed that ChatGPT consistently failed to identify the relevant scripts and often misallocated importance, frequently overemphasising noun phrases or relying on familiar patterns from its training data. These errors mirror those seen in tasks requiring complex semantic reasoning, such as the Adversarial NLI dataset (Nie et al., 2020). There were further difficulties in spatial and physical reasoning, consistent with Borji (2023). Thus, the hypothesis was falsified. However, the study successfully identified specific shortcomings of ChatGPT, which may inform future work on the model.