On the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency?

Large language models (LLMs) are increasingly used in decision-making contexts, yet their ability to reason over event structure-an important component in the situational awareness needed to make complex decisions-is not well understood.By operationalizing protorole theory, which characterizes agents via properties such as instigation and volition and patients via properties such as change of state, we examine the ability of LLMs to answer questions that require complex, multi-step event reasoning.Specifically, we investigate the extent to which LLMs capture semantic roles such as "agent" and "patient" through zero-shot prompts, and whether incorporating semantic proto-role labeling (SPRL) context improves semantic role labeling (SRL) performance in a zero-shot setting.We find that, while SPRL context sometimes degrades SRL accuracy in high-performing models (e.g., , it also uncovers an internal consistency between SPRL and SRL predictions that mirrors linguistic theory, and provides evidence that LLMs implicitly encode consistent multi-dimensional event role knowledge.Furthermore, our experiments support prior work showing that LLMs underperform human annotators in complex semantic analysis.