Evaluating an LLM Pipeline for End-to-End MCDA

Multi-Criteria Decision Analysis (MCDA) requires human involvement at every stage of its process. This paper proposes a systematic LLM evaluation protocol for end-to-end MCDA pipelines, using published expert rankings as a benchmark. In addition, it implements a four-stage LLM pipeline as a case study applying the protocol to three leading LLMs across five benchmark problems selected to vary in application domain and in the number of alternatives. The protocol comprises 225 independent pipeline executions (3 models × 5 repetitions × 3 temperatures × 5 problems); each execution applies three MCDA methods (LLM-weighted linear scoring, TOPSIS, PROMETHEE II) to the same LLM-generated score matrix, yielding 675 method-specific outputs in total. Consistent with a smaller pilot study of this protocol, differences between models are small relative to problem identity (Kruskal–Wallis, H = 1.10, p = 0.577, η2 = 0.005; a small but significant effect emerges once problem identity is controlled for, η2 = 0.6%), while problem identity remains the dominant performance factor—and a larger one than previously observed (η2 = 0.761, H = 170.51, p < 0.001, N = 225 executions). Kendall’s τ ranges from −0.34 on a ten-alternative problem to 0.56 on a four-alternative problem; a permutation test against random rankings indicates that two of the five benchmark problems produced agreement below chance level, while two problems produced agreement above chance level; since the test treats executions as independent, these deviations should be read as descriptive rather than inferential. Top-3 Accuracy ranges from 10.1% to 72.1% across problems, relative to chance baselines of 30% to 75%. Reliability varies strongly across problems, and the compensatory/non-compensatory distinction from the pilot does not fully explain this. In this pipeline, the scoring stage receives the names of the alternatives and criteria but not the numerical decision data, so the results reflect the LLM’s prior knowledge; its role as an explanatory factor is proposed as a hypothesis for future testing.