Natural Language Reasoning in Large Language Models: Analysis and Evaluation

While Large Language Models (LLMs) have demonstrated promising results on a range of reasoning benchmarks-particularly in formal logic, mathematical tasks, and Chain-of-Thought prompting-less is known about their capabilities in unconstrained natural language reasoning.Argumentative reasoning, a form of reasoning naturally expressed in language and central to everyday discourse, presents unique challenges for LLMs due to its reliance on context, implicit assumptions, and value judgments.This paper addresses a gap in the study of reasoning in LLMs by presenting the first largescale evaluation of their unconstrained natural language reasoning capabilities based on natural language argumentation.The paper offers three contributions: (i) the formalisation of a new strategy designed to evaluate argumentative reasoning in LLMs: argument-component selection; (ii) the creation of the Argument Reasoning Tasks (ART) dataset, a new benchmark for argument-component selection based on argument structures for natural language reasoning; and (iii) an extensive experimental analysis involving four different models, demonstrating the limitations of LLMs on natural language reasoning tasks.