Large Language Models for Requirement Compliance Checking : A framework for road construction

Background: Requirement Engineering process is time consuming and very me-ticulous. Engineers may spend days trying to make sure requirements are struc-tured and worded correctly to avoid misinterpretation. To streamline the processthe Swedish Transport Administration in collaboration with IBM have developed aproof-of-concept system (Dokument-AI) for requirement compliance checking whichleverages an LLM to modify the input requirement to align with a pre-defined set oflinguistical and structural rules. However, being a proof of concept system which alsoleverages an LLM, it has to undergo an evaluation in order to assess its performance.Objectives: The objectives of this thesis are to create a framework for evaluatinghow Dokument-AI performs its given task, specifically we aim to measure precision.Method: The method consists of using LLM-as-a-Judge to judge whether the newproposed requirement follows the grammatical and linguistical requirements set byDokument-AI, in addition to that we use BLEU and chrF to measure how similarthe new requirement is to the original requirement.Results: The results show that Dokument-AI managed to generate requirementsthat passed in 75.5% of the total cases and from the manual review we can see thatLLM-as-a-Judge should be accurate in 95.2% of the cases.Conclusions: In conclusion, while Dokument-AI shows considerable promise for fastevaluation of potential new requirements, with re-formulations passing in 75.5% ofcases, its accuracy is not sufficient for critical applications. Although it can providevaluable reasoning and insights, even when its judgments are disputed, the risk ofinaccuracies is too high for the requirements engineering process, where mistakes canhave long-term negative impacts on a project. However, if used in conjunction withhuman supervision, the system’s performance can be drastically improved, makingthe data far more reliable and usable for practical applications.