MaViLa: Unlocking new potentials in smart manufacturing through vision language models

In smart manufacturing, there remains a gap in the system-level understanding of manufacturing processes that hinders the effective integration of artificial intelligence (AI) for autonomous planning and execution in dynamic real-world scenarios. This paper presents MaViLa, an advanced vision language model (VLM) specifically designed for the smart manufacturing domain. MaViLa enhances visual understanding in the manufacturing domain through two key approaches: first, it uses a retrieval augmented generation (RAG) pipeline to incorporate domain knowledge during dataset creation, and second, it implements a robust two-stage training paradigm of pre-training followed by instruction fine-tuning. Comparative evaluations of domain-relevant benchmarks demonstrate MaViLa’s superior performance over general-purpose VLMs, particularly in manufacturing-specific tasks such as process optimization and quality control. Experimental results, including laboratory tests and in-situ monitoring applications, highlight the effectiveness of MaViLa in scene understanding and decision-making support. With its scalability and seamless integration of external tools, MaViLa paves the way for more efficient human–machine interactions and the development of autonomous, holistic manufacturing systems. These advancements establish MaViLa as a key technology that unlocks new potential for smart manufacturing. • Introduced MaViLa, a vision language model specialized for the manufacturing domain. • Achieved state-of-the-art performance in the manufacturing domain. • Showcased MaViLa’s potential as an advanced AI agent in autonomous manufacturing.