Did You Understand the Problem? – Evaluating Fine-Tuned Language Models for Identifying AI-Related Tasks in Product Development

Abstract Recent advancements in machine learning (ML) offer significant potential to support product development. However, adoption in industry remains limited due to challenges in the initial step of translating domain-specific product development problems into ML-related tasks. This process requires domain-specific knowledge in both product development and ML, which is often lacking. Prior research investigated the use of large language models (LLMs) as recommender systems for suggesting suitable ML algorithms based on product development-related problem formulations. However, the results indicated only moderate success, primarily due to a lack of understanding of the underlying product development tasks, which often led to incorrect identification of the corresponding ML problem types. Therefore, this study investigates the use of small, fine-tuned language models (LMs) for classifying product development-related problem formulations into ML problem types as a preliminary step toward identifying suitable ML algorithms for the respective task. A curated dataset of 400 problem statements, extracted from abstracts of peer-reviewed publications identified through a systematic literature review, was expanded to 2,400 entries using lexical and semantic data augmentation. This dataset was subsequently used to fine-tune and evaluate both encoder-only and decoder-only models. The findings indicate that decoder-only models surpass encoder-only models in performance, reaching F1-scores of up to 83.9% on the augmented dataset. However, performance drops significantly when controlled data splits are applied, indicating a tendency toward overfitting rather than generalization. Future research should focus on expanding the dataset and exploring prompt engineering techniques to enhance model performance.