Benchmarking and fine-tuning large language models for domain-specific materials science: a case study toward GPU application

Despite significant progress in large language models (LLMs) with general reasoning capabilities, their adaptability remains limited in interdisciplinary domains that require specialized expertise, such as those related to cutting-edge chip fabrication in materials science. To address this gap, this study constructs a domain-specific benchmark for graphics processing unit (GPU) materials science, comprising a systematically constructed evaluation dataset and a comprehensive assessment of representative LLMs on domain-specific tasks. As part of this benchmark development, we also develop GPUmatLLM, a specialized domain language model fine-tuned on high-quality data from academic papers, patents, and other relevant sources for GPU materials science. Benchmarking results reveal a notable performance gap among the representative LLMs, with no single model excelling in both general scientific reasoning and GPU material-specific knowledge acquisition. In contrast, GPUmatLLM demonstrates improved comprehensive performance on GPU materials tasks, surpassing baseline models in domain-specific inference while maintaining competitive performance in general scientific reasoning. This work provides an effective framework and practical paradigm for benchmarking and optimizing domain-specific LLMs in specialized materials science fields.