Locating requirements in backlog items: Content analysis and experiments with large language models

As agile development has become mainstream, requirements are increasingly managed via issue tracking systems (ITSs). These systems provide a single point of access to the product and sprint backlogs, bugs, ideas, and tasks for the development team. ITSs do not clearly separate requirements from work items. We first tackle a knowledge problem concerning how requirements are formulated in ITSs, including their categoriziation and granularity, the presence of multiple requirements, and the existence of a motivation. Second, to assist practitioners in finding requirements in poorly organized ITSs without changing their way of working we investigate the potential of automated techniques for identifying and classifying requirements in backlog items. Through quantitative content analysis, we analyze 1636 product backlog items sampled from fourteen projects. To explore automated techniques for identifying requirements, we experiment with large language models (LLMs) due to their recent significance in NLP. The labeling of backlog items is largely inconsistent, and user-oriented functional requirements are the prevalent category. A backlog item often contains multiple requirements with different levels of granularity. The experiments with LLMs reveal that encoder-only models (BERT and RoBERTa) are most suitable for extracting and classifying requirements in backlog items compared to decoder-only models (Llama 3, Mistral 7B and ChatGPT with GPT 4). We reveal knowledge and patterns about requirements documentation in ITSs, leading to a better empirical understanding of Agile RE. The experimental results with LLMs provide the foundation for developing automated , unobtrusive tools that identify and classify requirements in ITSs. • We perform content analysis of 1636 product backlog items sampled from 14 projects. • Backlog items are labeled inconsistently, making locating requirements difficult. • We apply five LLM architectures for identifying and classifying requirements in backlog items. • We compare the performance of encoder-only and decoder-only neural architectures. • Encoder-only models are most suitable for locating requirements in backlog items.