Data-Centric Fine-Tuning of Small Language Models for Automatic Extraction of Technical Requirements

Training small language models for specific tasks often encounters a significant challenge: the limited availability of high-quality labeled data, which can restrict model performance. This constraint is especially critical in organizations handling sensitive data, where large language models cannot be readily deployed due to privacy concerns, high costs, and dependency on external providers. While existing research demonstrates the effectiveness of large language models in automating text-based tasks, there is a lack of methods tailored to fine-tuning small language models for secure, domain-specific applications with minimal labeled data. To address this gap, we introduce a data-centric fine-tuning method that systematically enhances training data through prompt engineering, making small language model fine-tuning feasible and effective in data-constrained environments. This study evaluates the proposed method on a real-world use case with an industry partner, focusing on the automatic extraction of technical requirements from full-text documents and using both quantitative metrics and qualitative expert assessments. Our findings reveal that the fine-tuned small language model achieves accuracy and consistency comparable to human service providers, while outperforming baseline models, including GPT-4-turbo, on key evaluation metrics. These results underscore the potential of data-centric fine-tuning for adapting small language models to high-stakes, privacy-sensitive tasks, offering a scalable alternative to LLMs enabling their applications to real-world use cases.