Who Should Test the Requirement? A Comparative Study on Requirements Classification for Assigning Test Teams using the Pre-Trained Models

Analyzing software requirements is a complex task, particularly for projects with a large volume of requirements, and when conducted manually, this task is time-consuming and prone to human errors. Moreover, once the software implements the requirements, it is essential to conduct tests to ensure the correct validation. Within a software institute, each new requirement can be assigned to a test team (teams 1 and 2) responsible for ensuring coverage by updating or creating test cases. There are instances in which a requirement is not assigned to either the team or is assigned to both. Each test team is tasked with validating a specific scope of requirements, making it crucial that each requirement is analyzed and validated by an appropriate test team. If a test team fails to validate a requirement within its scope, it can result in software vulnerability. To mitigate these issues, this paper described the use of pre-trained models, such as BERT, XLNet, and ELECTRA, to automate the process of requirement classification, thereby determining which test team should validate each new requirement. We compared the models based on accuracy, precision, recall, F1-Score, and Area Under Curve (AUC) macro metrics. Notably, the XLNet model demonstrated superior performance among the models, achieving 93.16% AUC Macro, while the BERT model achieved 91.28% AUC, and the ELECTRA model achieved 90.17% AUC.We also applied the non-parametric Friedman test to statistically validate the results, followed by the Conover squared rank test, with a significance level of 0.05. The results indicate that XLNet outperformed the BERT and ELECTRA models, exhibiting a superior capacity for assigning requirements to the correct test teams. Given the promising results of this research, this study aims to demonstrate the viability of using pre-trained models as a solution to optimize the testing process in the software industry.