Source Expertise, Argument Strength, and Multidimensional Source Sensitivity in Large Language Model Evaluation

Abstract Large language models are increasingly used to evaluate arguments, responses, and decisions, yet their judgments may be influenced by contextual information about who produced the content. This study examined how source expertise relevance and argument strength affect LLM evaluation and whether source effects differ across judgment dimensions. Twenty arguments spanning 10 topics were constructed as strong–weak pairs and presented under four source conditions: relevant expert, irrelevant expert, non-expert, and no source. Four large language models evaluated each argument on logical quality, credibility, persuasiveness, and agreement. Five repetitions were collected for each model-by-argument-by-source-by-dimension combination and averaged before inferential testing. Human validation assessed the intended argument-strength manipulation but showed negligible overall strong–weak separation in the second validation round, including two reversed topic pairs. Separate manipulation checks indicated that the models rated topic-relevant expertise and general professional expertise differently across source conditions. Source attribution most consistently affected credibility. Topic-level robustness analyses showed significant source-condition effects on credibility across all four models, while persuasiveness remained significant only for GPT-5.6 Sol. Relevant-expert attribution did not produce a universal advantage: credibility was significantly higher for relevant than irrelevant experts for Claude Opus 5.5 and DeepSeek Flash, and Claude Opus 5.5 also showed significantly higher persuasiveness for relevant than irrelevant experts. Exploratory analyses suggested that this pattern may reflect an irrelevant-expertise penalty more than a universal bonus for relevant expertise. No robust evidence showed that relevant expertise disproportionately improved weak arguments. Credibility was descriptively more source-sensitive than logical quality, but corrected direct contrasts were not statistically significant. Overall, LLM evaluation was neither fully source-neutral nor characterized by simple deference to experts. Source expertise produced dimension-specific and model-dependent effects, with credibility showing the clearest sensitivity.