Conscious Distilling: Small Models Achieve Strong Performance in Math QA Tasks

Large language models have the ability to solve certain mathematical problems. However, using existing large language models requires enormous computational resources, and they perform poorly when faced with problems that require mathematical reasoning. To address these issues, we propose an efficient mathematical knowledge distillation scheme called Conscious Distilling. Our proposed scheme follows these steps: Layer Removal Technique: We remove the layers of the original model that have the least impact on the domain, allowing the model to retain more relevant information while remaining lightweight. Conscious Distilling Method: We intentionally guide the model to focus more on the analysis of the process rather than just the answer. Specifically, this strategy reduces the Qwen2 model from 7 B to 3.2 B, achieving accuracy improvements of 4.2% and 7.2% on the MATH and GSM8K datasets, respectively, compared to the original model.