BOLT: Building Open-Source LLMs for Your Target Domain via Automated Hierarchical Knowledge Distillation

Adapting open-source large language models (LLMs) to specialized domains remains a critical challenge due to domain knowledge gaps, data scarcity, and reference hallucination. Existing approaches often neglect the structural characteristics of domain knowledge and fail to provide principled estimations of knowledge scope, resulting in data homogenization and suboptimal adaptation, while leaving reference hallucination unmitigated. This paper introduces BOLT(Building Open-source LLMs for your Target domain), a modular end-to-end framework that tailors open-source LLMs for domain-specific scenarios. BOLT systematically estimates domain scope, constructs structured hierarchical knowledge trees, distills diverse and semantically aligned training data from advanced teacher LLMs, and employs curriculum learning for progressive model optimization. To address reference hallucination, BOLT substitutes generative methods, which are susceptible to hallucinations, with a matching-based strategy, thereby alleviating the problem and significantly improving reference recommendation accuracy. Extensive experiments across diverse domains and models demonstrate that BOLT enables the efficient modeling of structured hierarchical domain knowledge and effectively enhances reference recommendation accuracy while preserving both training efficiency and robustness throughout the adaptation process.