How Does Memorization Impact LLMs' Social Reasoning? An Assessment using Seen and Unseen Queries

As Large Language Models (LLMs) have rapidly advanced in social reasoning tasks, their applications have expanded to domains such as healthcare and psychology. Given the direct interaction of users with these applications, it is essential to evaluate the performance of LLMs, particularly in human-like social reasoning capabilities. While previous studies have explored human-aligned social reasoning in LLMs, they have not adequately assessed whether the generated reasoning answers stem from the LLMs' memorization of training data or their natural language understanding. In this study, we aim to address this gap by assessing the impact of training data memorization on the human-aligned social reasoning capabilities of LLMs. We introduce IR+CoT (Information Retrieval (IR) + Chain of Thought (CoT)), a framework that leverages retrieved information from input questions to fine-tune prompt templates and employs CoT methods. IR+CoT mitigates the effects of memorization and enhances the LLMs' social reasoning performance. Experiments on three LLMs, using seen (present during the training of the LLMs) and unseen (introduced post-training) questions from Reddit and Lemmy, show that IR+CoT enhances social reasoning and reduces memorization effects. This research's novelty lies in using old and new questions to assess memorization's impact on social reasoning.