ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study

Recent advances in language modeling demonstrate the need for high-quality domain-specific training data, especially for tasks that require specialized knowledge.General-purpose models, while versatile, often lack the depth needed for expert-level tasks because of limited domain-specific information.Domain adaptation training can enhance these models, but it demands substantial, high-quality data.To address this, we propose ORBIT, a costefficient methodology for curating massive, high-quality domain-specific datasets from noisy web sources, tailored for training specialist large language models.Using astronomy as a primary case study, we refined the 1.3T-token FineWeb-Edu dataset into a highquality, 10B-token subset focused on astronomy.Fine-tuning LLAMA-3-8B on a 1Btoken astronomy subset improved performance on the MMLU astronomy benchmark from 69% to 76% and achieved top results on As-troBench, an astronomy-specific benchmark.Moreover, our model (Orbit-LLaMA) outperformed LLAMA-3-8B-BASE, with GPT-4o evaluations preferring it in 73% of cases across 1000 astronomy-specific questions.Additionally, we validated ORBIT's generalizability by applying it to law and medicine, achieving a significant improvement of data quality compared to an unfiltered baseline.We open-source the ORBIT methodology, including the curated datasets, the codebase, and the resulting model at https://github.com/ModeEric/ORBIT-Llama.