A large language model for traditional Chinese medicine based on Llama3

This project focuses on fine-tuning Llama 3 and other models such as Phi-3.5 and Mistral on Traditional Chinese Medicine (TCM) datasets. While Llama 3 is powerful in general NLP tasks, it is not optimized for Chinese, especially in the context of TCM. To address this, we fine-tune the models on TCM-specific data to enhance their understanding of herbal names, diseases, and treatments, especially since these proper nouns are often in classical Chinese in ancient China. We go beyond Chinese and include classical Chinese datasets. During fine-tuning, we use Unsloth to speed up the fine-tuning process and introduce LoRA (Low Rank Adaptation) to reduce computational cost. The TCM dataset is formatted for Hugging Face and labeled according to the requirements of each model. Models are evaluated using BLEU score, perplexity, memory usage, and training time. Results show that Llama 3 achieves the best overall performance, with significant improvements in BLEU score and perplexity, while maintaining moderate memory usage. Phi-3.5 and Mistral also show improvements, with Phi-3.5 being the most memory efficient and Mistral excelling in handling large-scale data.