Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively

Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks.However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost.To address this, we first introduce the 3E Criteria to systematically assess the costeffectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains.To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies.Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs' internal self-evaluation.And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps.We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains.Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.Q: Josh decides to try flipping a house.He buys a house for $80,000 and then puts in $50,000 in repairs.This increased the value of the house by 150%.How much profit did he make?Q1: How much did Josh spend on repairs?A1: Josh spent $50,000 on repairs because that was the amount he put into fixing up the house.The answer is $50,000.Q1: How much did the value of the house increase after the repairs?A1: The value of the house increased by $120,000 after the repairs.The answer is $120,000.