MetaBPL: Fault Detection in Business Logic Systems

Critical workflows in manufacturing, infrastructure, and logistics rely heavily on business process logic systems, where even minor faults or vulnerabilities can lead to significant operational disruptions or security breaches. For large companies, a typical product recall may cost more than ten million USD, and every hour unplanned downtime of a manufacturing line might incur a million dollars in losses. While typical processes like statistical quality assurance and auditing can help mitigate future occurrences of faults, they time consuming and often lead to downtime as corrective actions are put in place. To address these real-world challenges, we deploy a Large Language Model (LLM) agent capable of detecting vulnerabilities and faults within business systems. Our dual-pipeline framework first extracts rules and specifications, which are efficiently stored for use in the subsequent fault detection pipeline. By integrating advanced natural language processing (NLP) techniques and Retrieval-Augmented Generation (RAG) methodologies, our approach automates the extraction and analysis of process specifications from diverse document formats, including BPMN diagrams, structured PDFs, and JSON files. Once extracted and organized, these rules and their instantiations can be analyzed by the system’s fault detection pipeline or exported to formal languages for evaluation. The first pipeline automates specification generation through advanced text extraction methods. Utilizing optical character recognition (OCR), structured data extraction, named entity recognition (NER) and semantic processing, we identify and map key phrases to form fields using domain-specific lexicons. We employ coreference resolution to establish accurate mappings across documents, allowing us to automatically generate consistent specifications. We convert the extracted data into a vector database (ChromaDB) to facilitate similarity-based retrieval. Building upon this foundation, we implement the second pipeline using a RAG architecture designed for fault detection. When a user submits a query, the system searches the vector database to retrieve the relevant sub-context, which is combined with a business logic-specific prompt derived from the user's query. Dynamically generated fault detection queries enable the LLM to identify discrepancies in workflows and produce detailed fault analyses and corrective recommendations. We evaluate our approach using three metrics: (1) the correctness of the LLM-generated answers (completeness versus hallucination), (2) question-answer accuracy given the context provided to the LLM, and (3) the quality of the RAG process. Our corpus for evaluation is based on artifacts related to typical business processes such as excerpts from regulations (DOT), standards (ISO-9000), and organizational documentation such as Quality Maintenance System and internal business form (travelers, work instructions, bills of material etc.). Our dual-pipeline framework enhances the precision and scalability of fault detection, offering advancements in the automated optimization of business process logic systems. Also, it empowers end users to verify the correctness of processes without extensive technical expertise and mitigate risks associated with faulty business logic. By improving the efficiency of critical workflows, our solution contributes to the overall security and robustness of essential industrial operations.