Evaluating cost-effectiveness and coherence of LLMs for supplement recommendations using routing techniques

Large Language Models (LLMs) have demonstrated considerable potential for the task of personalized recommendation, yet it remains difficult to ensure that they are cost-effective, consistent, and reliable. In this paper, we compare systematically six different LLMs for the task of recommending supplements on a dataset of 50 anonymous health records. To achieve best performance with low computational cost, we implement routing mechanisms in which tasks are routed to models dynamically based on their strength and efficacy. We measure self-coherence — coherency between a model’s outputs — and cross-coherence — agreement between models — in terms of statistical metrics like intra-class correlation, Kendall’s W, and Spearman’s rank correlation. Model differences are contrasted using Kruskal-Wallis and post-hoc Dunn’s tests. Our findings illustrate trade-offs among quality, consensus, and computational expense and offer insights for the deployment of LLMs in cost-constrained clinical environments.