Deploying Vision Retrieval Augmented Generation as Assistant for xPPU Maintenance

The maintenance and operator training of manufacturing systems faces challenges due to the overwhelming volume and system-specific documents, which require extensive memorization. While large language models have shown the capability to support operators their lack of knowledge of machine specific information limits their out-of-the-box applicability. To address this, Retrieval-Augmented Generation has been introduced as a solution for large-scale document retrieval in various industries, including automation and manufacturing. However, processing visual elements such as images, graphs, and diagrams within these documents remains a significant hurdle. This paper proposes the integration of Visual Retrieval-Augmented Generation (VRAG) into a maintenance support system for the lab-sized manufacturing systems extended Pick and Place Unit (xPPU). The deployed VRAG combines advanced Chain-of-Thought (CoT) reasoning. The system's evaluation framework is designed to assess performance across various stages, optimizing both indexing and retrieval. Results demonstrate that VRAG significantly improves maintenance efficiency, providing a robust solution for handling a wide range of queries that appear in typical maintenance tasks. The contributions of this work include the successful application of VRAG to the xPPU system, the development of comprehensive evaluation metrics involving experts familiar with the xPPU system, and the integration of agentic capabilities for improved query resolution.