Combined hidden-Markov model and Siamese network approach for assembly operation recognition and error detection

The growing demand for customised products is driving manufacturing towards more flexible production systems. In this context, manual assembly lines remain essential for complex or small-batch products, as human cognitive and advanced manipulation capabilities enable better adaptation to changes in production processes. However, the experience and working conditions of human operators significantly impact the quality of the final product. External factors such as stress, fatigue, and cognitive overload can lead to errors, especially in flexible environments that require rapid adaptation. Monitoring human activity to detect errors during the process is, therefore, crucial. This paper presents a Siamese network-based approach to recognise assembly steps and detect errors, easily adaptable to different assembly scenarios. By comparing images captured during the process with a reference dataset, the Siamese Neural Network (SNN) identifies the last assembly step performed and possible discrepancies. The SNN is integrated into a Hidden Markov Model framework to improve recognition accuracy and process reliability. Experimental validation demonstrated the model's adaptability to various assembly scenarios, including a real-world use case of industrial packaging, without the need to retrain the model. The method also proved to be robust to different working conditions and human variability.