“Sorry, I Cannot Fulfill That Request”: Analyzing Large Language Model Responses, Redirections, and Refusals to Polarized News Topics

ABSTRACT We are reaching an era where the public increasingly relies on large language models (LLMs) for information on current events. Existing research on the subject asks LLMs to take a political stance through survey questionnaires, persona adoption, or multiple‐choice prompting. The following research examines the implicit political lean of LLMs when responding in natural language to queries on 77 topics that were of public interest from 2017 to 2021. Four LLMs were prompted using two natural prompting styles, resulting in 808 unique responses. Responses were human annotated to identify topics that LLMs redirected or outright refused to answer, and were classified via a neural network as conservative, moderate, or liberal. Further, LLM responses were analyzed on paradigms of whether topics were polarized, international, or asked using non‐neutral language. Findings suggest that these LLMs lean moderate to liberal, erroneously refuse neutral topics, and are inconsistent in answers to the same prompts. These findings illustrate the risk of relying on generative AI for answers in an increasingly polarized environment and call for information professionals to examine and discuss implicit misinformation in the age of AI.