Large Language Model-Based Extraction of Logic Rules from Technical Standards for Automatic Compliance Checking

In this research, we design logic rules as a representation of technical standards documents related to ship design, which will be used in automatic compliance checking. We present a novel design of logic rules based on a general pattern of technical standards’ clauses that can be produced automatically from text using a large language model (LLM). We also present a method to extract said logic rules from text. First, we design data structures to represent the technical standards and logic rules used to process the data. Second, the representation of technical standards is produced manually and tested to ensure that it can give the same conclusion as human judgment regarding compliance. Third, a variation of prompting methods, namely pipeline method and few-shot prompting, is given to LLM to instruct it to extract logic rules from text following the design. Evaluation against the logic rules produced shows that the pipeline method gives an accuracy score of 0.57, a precision of 0.49, and a recall of 0.62. On the other hand, logic rules extracted using few-shot prompting have an accuracy score of 0.33, precision of 0.43, and recall of 0.5. These results show that LLM is able to extract a logic rule representation of technical standards. Furthermore, the representation resulting from the prompting technique that utilizes the pipeline method has a better performance compared to the representation resulting from few-shot prompting.