KDLLM: Copyright-Preserving LLM based on Knowledge Distillation

Large Language Models (LLMs) have emerged as the cornerstone of various natural language processing activities, enabling everything from chatbots to text classification and summarization. However, using LLMs presents some significant challenges, most notably the threat of intellectual property infringement from the exposure of the entire model and the excessive communication and storage overhead associated with their large size. We propose KDLLM, a novel knowledge distillation-based framework for efficient and compact LLMs to address these challenges. KDLLM transfers the performance of a large teacher LLM to a significantly smaller student model with high performance similarity to its teacher, while obscuring architectural and parameter-level details to protect the intellectual property of the original model. The resulting student model substantially reduces the memory footprint and transmission overhead, making it amenable to deployment in bandwidth-constrained or security-sensitive environments. Comprehensive experiments demonstrate that KDLLM achieves robust performance preservation and boosts copyright protection and communication efficiency.