LLM-Based Agentic AI in Manufacturing: A Scoping Review of Applications, System Designs and Evaluation Practices

Large language model (LLM) agents are increasingly used in manufacturing to design products, plan production, diagnose faults and operate equipment, but their designs and evaluations remain difficult to compare. We analyse 164 studies identified through Scopus, Web of Science and IEEE Xplore to examine manufacturing applications, agentic system designs and evaluation practices, and to identify where evaluation coverage falls short of the evidence that these task and design types would require. We propose taxonomies of applications, agentic designs and evaluation practices, and relate task and design characteristics to the evidence required to assess them. The findings reveal an imbalance between assessing output quality and verifying whether outputs satisfy manufacturing requirements. For planning and decision-making systems, executability and constraint compliance receive limited evaluation, while most evidence comes from offline data or simulation. Limited assessment of safety, stability across repeated runs and performance in industrial settings leaves a gap between demonstrated task performance and evidence of reliable operation under manufacturing conditions. The resulting taxonomies and evidence requirements help researchers select evaluations suited to each manufacturing use case and distinguish demonstrated task performance from evidence of reliable operation.