Ontology-Based NLP Tool for Tracing Software Requirements and Conceptual Models: An Empirical Study - Raw data, textual descriptions, experimental objects, and scripts

This repository contains all raw data, textual descriptions, experimental objects, and scripts from the paper "Effect of an Ontology-Based NLP Tool forTracing Software Requirements and Conceptual Models: An Empirical Study of OntoTraceV2.0", currently under review at the Requirements Engineering Journal. How to use: In the following bullet points, we describe each file you will find in this repository. Experimental-Object-Text-Description.pdf It contains the textual description of the use case where we extracted the user stories and Merlin models and gave them to the subjects as an experimental task. This textual description was provided by the authors of the high-cited work in https://doi.org/10.1016/j.infsof.2015.02.012 based on the transport management system. Merlin-Model.mxp Contains the .mxp file representing the MERLIN model that subjects faced as experimental tasks. To open and explore it, you must load the files on the https://www.merlin-academic.com webpage. UserStories.docx It contains a set of seven user stories to be traced manually and with OntoTraceV2.0. KQ-QE-A.csv; KQ-QE-B.csv; and KQ-QE-C.csv It contains the knowledge questionnaire results with scores from 0 to 9 (anonymized). Questions are in random order, but here we have a set of the questions presented to the subjects: Q: What are the 3 main components of a user story? Q: Select all options that fit a user story structure. Q: Select an option that contains a User story that describes the functionality highlighted on the mockups- Q: Select the option that contains a correct description of the diagram: Q: The Aircraft Reviewer can create 0 to several reviews. The review, in its turn, can only have 1 Aircraft Reviewer. Considering the description, is this data model correct? Q: What would the "Mothership ID" and the "Mothership name" be to the "Mothership" class? Q: Traceability in Software Development is creating links only between software requirements and test cases Q: An Artifact can have multiple trace links (True/False) Q: A Target Artifact can take a form of ... Demo-QE-A.csv; Demo-QE-B.csv; and Demo-QE-C.csv The files contain the raw demographic data from subjects in QE-A, QE-B, and QE-C. You will find the code for each subject, study program, experience in software development, traceability, testing and industry, as well as the industry where they work. Data-QE-A.csv; Data-QE-B.csv; and Data-QE-C.csv Contains the raw data from each QE with the following columns: ID. Numeric ID that we assigned to each experimental subject. Method. Column to differentiate if they use a "Manual" strategy or "Tool" strategy (OntoTraceV2.0). Problem. Column to determine the experimental object each subject faced (O3 all of them). Starting time. Starting time executing the experimental task. Ending time. The time when the subject completed the experimental task. Q1-Q14. Likert-scale questions from 1 to 5 about satisfaction questions. You can find each question in the G3-S1-Guidelines and G3-S2-Guidelines. We assign the ID in order of question. The Number of correct discovered traceability links (TP; True Positive). Generated trace links that fit with what we expected to be traced based on the experimental object design. The Number of wrongs discovered traceability links (FP; False Positive). Generated trace links that do not fit with what we expected to be traced based on the experimental object design. The Number of missed correct traceability links (FN; False Negative). Not generated trace links that fit with what we expected to be traced based on the experimental object design. The Number of missed wrong traceability links (TN; True Negative). Not generated trace links that do not fit with what we expected to be traced based on the experimental object design. Precision. This column contains the precisions as (TP / (TP + FP)) in percentage. Efficiency. This column contains the efficiency as (#Number of traces / (Ending time - Starting time) in Traces/Min unity. PEU, PU, ITU AVG. On average, these columns contain the satisfaction measurements (Perceived ease of use, Perceived usefulness, and Intention to use). PEU, PU, ITU MED. These columns contain the satisfaction measurements (Perceived ease of use, Perceived usefulness, and Intention to use) with the Median value. Statistic-model-2025.py It contains the Python script to calculate the Wilcoxon-Mann-U test. G3-S1-Guidelines and G3-S2-Guidelines PDF files containing an example of instructions delivered to subjects as experimental task for manual and tool-supported trace link discovery with OntoTraceV2.0 containing also the satisfaction questionnaire. Experimental-Object-Design.xlsx Contains the experimental object design with the following tabs: UserStories. This tab contains all user stories in the experimental object. That includes the ID, the source where it is taken from (based on the textual description), user role, action, objects, reason, and the final result concatenating the last four columns. Sources. This tab contains all source artefacts based on the previously defined user stories. That includes the user story ID, source ID, name, and type (Role, Action, Object, or Goal). Targets. This tab contains all target artefacts based on the Merlin EDG model. That includes the target ID, name, and type (Class, Relationship, Method, Attribute). UserStroiesAndRulesClass. This tab contains all rules used to create the base solution of traceability. We use transformation rules and our experience transforming user stories into EDG models to devise a base traceability solution. We link each trace into a rule based on literature. This tab includes the element, rules based on Nasiri et al. 2020 or Barigolvski et al. 2022, resulting element, target id, target, and target type. Traces. This tab summarizes all traces found in the experimental object using the previously defined source and target IDs. Traces_interpeted. This tab interprets the Traces tab to create a more readable view of the traces in the experimental object. Nasiri2020. This tab contains the rules based on Nasiri et al. 2020 to transform user stories into EDG models (or UML-like-class-diagram models). That includes a Rule column and a description of the rule. Barigolvski2022. This tab contains the rules based on Barigolvski et al. 2022 to transform user stories into EDG models (or UML-like-class-diagram models). That includes a Rule column, a description of the rule, the expected outcome, and a set of remarks. Traceability-Ontology.owl It contains the traceability ontology for tracing user stories into EDG models together with the experimental object (empty version; i.e., without trace links). This file can be open usign any OWL editors, nevertheless we recomend using Protégé. We include some screenshots for visualizing the ontology into Protégé OWL Viz: View just with ontology classes (OntologyClasses.png) View with all individual instances (OntologyInstances.png) QE-B-FCG-#.jfif; QE-C-FCG-#.jpg; and FocusGroupTraceability.xlsx It contains the images of the focus group results from QE-B and QE-C. Moreover, we include the detailed clustering with the questions in the FocusGroupTraceability.xlsx file. Access to OntoTraceV2.0 We can provide access to the web-based OntoTraceV2.0 tool with the experimental object we introduced in the paper. Nevertheless, the IP of OntoTraceV2.0 is only accessible from devices located in Europe. WARNING: The website of OntoTraceV2.0 is not anonimized (authors names and afiliations apear in some pages). Therfore, to avoid violating the double-blinded peer-review process for the RE Journal, we decide to provide access only when strictely necesary by demand of the anonymous reviewers trought the editor, having in mind that the double-blind policy can be violeted. Nevertheless, for the sake of transparency, we include here some screenshots of OntoTraceV2.0 tool in execution (Anoynmized): Loging Web Page (OntoTraceLogin.png) OntoTrace Project View (OntoTraceProjectView.png) OntoTrace Traceability Overview (OntoTraceTraceabilityOverview.png) OntoTrace Specific Object to be Traced (OntoTraceSpecificObjectTraceabilityView.png) OntoTrace Specific Object Traced (OntoTraceTraceSpecificObjectTracedView.png)