Emails by LLMs: A Comparison of Language in AI-Generated and Human-Written Emails

The growing excitement around generative AI (and LLMs) is fueling a heightened interest in the development of AI-assisted writing tools.One popular context is AI-assisted email writing, and this paper explores how AI-generated emails compare to human-written emails.We obtained human-written emails from the W3C corpus and generated analogous AI-generated emails using GPT-3.5,GPT-4, Llama-2, and Mistral-7B, and compared AI-generated and humanwritten emails using a suite of natural language analyses across syntactic, semantic, and psycholinguistic dimensions.AI-generated emails are generally consistent across different LLMs but differ significantly from human-written emails.Specifically, AI-generated emails tend to be more formal, verbose, and complex, whereas human-written emails are often more concise and personalized.While AI-generated emails are slightly more polite, both types exhibit a similar level of empathetic tone in language.Further, we qualitatively examined user perceptions of AI and human-written emails by conducting a small survey of 41 participants and interviewing a subset of them.This study highlights preliminary insights into generative AI's distinct strengths and weaknesses in assisting email communication, and we discuss the theoretical and practical implications of the evolving landscape of AI-generated content.