Automated Knowledge Distillation Pipeline for Domain-Specific Small-Scale Language Models

Large Language Models (LLMs) have demonstrated notable advancements across a range of natural language processing tasks. However, the high costs of training and deploying these large models have intensified the interest in developing smaller-scale LLMs (sLLMs). Although sLLMs have the advantage of reduced training resources and serving costs, they typically exhibit lower performance due to their limited capacity. To overcome these limitations, we propose an automated knowledge distillation pipeline to develop domain-specific specialized sLLMs by leveraging the extensive capabilities of larger models. Our experimental results demonstrate that our approach significantly improves performance compared to the original sLLMs.