Evaluating the use of LLM-based multi-agent systems for internal information sharing and access within manufacturing

Abstract Internal information sharing in manufacturing remains constrained by data silos, heterogeneous systems, poor data quality, and limited interoperability across ISA-95 levels. While Large Language Models (LLMs) are increasingly proposed for manufacturing information access and for integrating information with operational technology, there is limited empirical evidence comparing their performance with unassisted human users on representative manufacturing tasks. This paper evaluates a modular LLM-based multi-agent system for retrieving and synthesising information across heterogeneous data sources from a 3D-printing factory demonstrator. The system was benchmarked against human participants on traceability and certification processing tasks involving packing lists, relationship-tracking records, machine logs, location-tracking data, and Authorised Release Certificates. Three task types under four data-quality conditions were tested, including: unaltered baseline data, whitespace injection, truncated identifiers, and severed relational links. Under the bounded experimental conditions, the highest-performing LLMs achieved higher mean accuracy and shorter completion times than the average unassisted participant, with Claude Sonnet-4 achieving 94.1% mean accuracy compared with 62.2% for humans. However, this advantage was conditional: some smaller models fell below the human average, and the best human participants outperformed all models. The system was robust to whitespace errors and severed relational links, but vulnerable to truncated identifiers and verbose prompt scaffolding. These findings indicate that multi-agent LLM systems can support bounded manufacturing information-access tasks as a decision-support layer, but require careful model selection, data validation, prompt design, and human oversight.