Communication styles and reader preferences of LLM- and human-authored COVID-19 information explanations: a case study

With the wide adoption of large language models (LLMs) in assisting professionals to create and share information, it is essential to examine their alignment with human communication styles and values in achieving communication effectiveness and acceptability. We situate this study within the context of explaining fact-checked health information, with a focus on communication styles because they influence whether explanations and corrections are understood, trusted, and acted upon. Recent studies have explored the potential of LLM-based fact-checking, but communication style differences between LLMs and human fact-checkers and associated reader perceptions remain under-explored. In this light, our study evaluates the communication styles of LLMs, focusing on how their explanations differ from those of humans in three core components of health communication: information linguistic features, sender persuasive strategies, and receiver value alignments. We compiled a dataset of 1498 misinformation claims and corresponding explanations from authoritative fact-checking organizations. Using this dataset, we employed chain-of-thought prompting with zero-shot and few-shot variations to generate LLM fact-checking responses to inaccurate health information. We drew from health communication theories and categorized communication styles along linguistic, persuasion, and value-based dimensions and measured how closely the LLM-generated responses aligned with professional explanations. Then, we examined human preferences with 99 participants who were unaware of LLM involvement and rated randomized fact-checking article pairs with switching orders. Our findings reveal that LLM-generated articles showed significantly lower scores in persuasive strategies, certainty expressions, and alignment with social values and moral foundations. However, human evaluation demonstrated a strong preference for LLM content, with over 60% responses favoring LLM articles for clarity, completeness, and persuasiveness. This reader preference of LLMs was driven by the structured presentation, clarity, and neutral tone, which may foster an impression of completeness and professionalism, even when the underlying content may lack nuance or rigor. Our results suggest that LLMs’ structured approach to presenting information may be more effective at engaging readers despite scoring lower on traditional measures of quality in health communication styles. This indicates both the potential and the limitations of utilizing LLMs in health communication or fact-checking workflows.