Personality expression and perception in large language models and humans

Abstract Growing numbers of people frequently turn to Large Language Models (LLMs) for concerns from various areas of life. Because its content has become ubiquitous, examining user perceptions of LLM-produced text and its distinguishability from human writing is crucial. This study investigated how seven LLMs (ChatGPT-4o, ChatGPT-3.5, Llama 3, Gemma, Gemini, Phi-3, Orca-2) describe themselves when compared to humans. LLM-generated outputs and human responses were collected using psychological questionnaires (Big Five, self-esteem, narcissism) and tailored prompts to elicit short self-descriptive texts. Six LLM and ten human judges then evaluated personality traits and text origin based on these self-descriptions (140 total, half LLM-generated). Results show that current LLM outputs can convey personality-consistent cues in questionnaires and texts. The perception of such significantly differed from that of humans. Both LLM and human judges accurately inferred personality from short self-descriptions and correctly differentiated between LLM-generated and human text. Linguistic analysis and participant feedback revealed textual cues distinguishing LLM-produced from human language and offered insights into how LLM versus human judgment and decisionmaking differ.