Federated Financial Reasoning Distillation: Training A Small Financial Expert by Learning From Multiple Teachers

Recent advancements in Large Language Models (LLMs) have significantly enhanced reasoning capabilities for complex tasks. However, the substantial computational demands and high deployment costs of these LLMs remain significant barriers to their widespread adoption. Although progress has been made in applying knowledge distillation to improve the performance of resource-friendly small LMs on general tasks, developing compact yet powerful financial reasoning expert LMs remains underexplored. In this paper, we present a comprehensive evaluation of leading reasoning and non-reasoning LLMs on their financial reasoning abilities by challenging these models to solve financial problems with detailed reasoning Chain-of-Thought (CoTs). Based on the benchmark results, we introduce a federated reasoning distillation framework where financial reasoning CoTs from various teacher LLMs are refined following "LLM-as-a-Judge" fashion to better supervise the training process of the student financial expert. Our experiments show that a 7B distilled LM can achieve competitive performance on two public finance question-answer datasets by learning from multiple teachers. To foster future research in financial AI, we will open-source the refined financial CoT dataset at https://github.com/yjump/FinCoT.