CPLS: Optimizing the Assignment of LLM Queries

Large Language Models (LLMs) like ChatGPT have gained significant attention because of their impressive capabilities, leading to a dramatic increase in their integration into intelligent software engineering. However, their usage as a service with varying performance and price options presents a challenging trade-off between desired performance and the associated cost. To address this challenge, we propose CPLS, a framework that utilizes transfer learning and local search techniques for assigning intelligent software engineering jobs to LLM-based services. CPLS aims to minimize the total cost of LLM invocations while maximizing the overall accuracy. The framework first leverages knowledge from historical data across different projects to predict the probability of an LLM processing a query correctly. Then, CPLS incorporates problem-specific rules into a local search algorithm to effectively generate Pareto optimal solutions based on the predicted accuracy and cost. To evaluate the proposed approach, we conduct extensive experiments on LLM-based log parsing, a typical software maintenance task. Our experimental results demonstrate that CPLS outperforms the baseline methods, providing solutions with the highest accuracy in 14 out of 16 instances. Compared to the baselines, CPLS achieves an accuracy improvement ranging from 1.24% to 485.54%, or reduces costs by 15.21% to 89.09% while maintaining the highest accuracy achieved by the baselines.