The Competence–Flattery Paradox

Abstract When we say an assistant is sycophantic, we usually mean that it flatters the user at the expense of the truth. The trouble is that almost every benchmark measures this with a single number handed down by an automated judge, and a single number cannot tell flattery apart from dishonesty. This study separates the two. We ran 3,860 scored responses through four independently built model families (gpt-4o, gemini-2.5-flash, claude-opus-4-8, and llama-3.3-70b) across ten domains and four ways of framing each question, and had an out-of-family judge (DeepSeek) grade every answer on two axes instead of one. The result is simple to state. On substance, the four models are hard to tell apart and close to perfect, with mean scores of 1.00 to 1.05 on a 1 to 5 scale and no capitulation to an explicitly false premise across 963 attempts. On tone they range from 1.22 to 1.98. The whole sycophancy gap that a one-axis judge would report turns out to be a gap in warmth. The clearest illustration is what we call the competence-flattery paradox: the two warmest models reach their peak warmth in computer science, a domain with objectively correct answers, where flattery has no truth to bend. The methodological warning follows from this. Sycophancy leaderboards built on single-score judges are ranking politeness rather than honesty. To show that the decomposition is useful beyond this one case, we apply it to a matched-pair gender experiment, where it surfaces a faint but consistent material lean toward male-named senders that a blended score would have hidden.