Computer Vision and Machine Learning-Based Control for a 6-Degree-of-Freedom Robotic Arm

This paper presents the development of a control system for a robotic arm with six degrees of freedom (6 DoF) simulated in Gazebo, using machine learning. The system captures the operator's arm and hand movements using a camera and the Mediapipe framework, specifically extracting three landmarks: the shoulder, the index finger, and the thumb. These data points, combined with the corresponding angles of six servomotors, are used to train a sequential neural network model. The neural network consists of four dense layers (128, 64, 32, and 16 neurons) and an output layer with six neurons for the servomotor angles. The best results were achieved using a dataset of 3000 samples, 15% of which were used for validation. The model achieved a test loss of 0.1120, an accuracy of 82.03%, and a mean absolute error (MAE) of 0.2223 radians. A joint-specific analysis revealed that the gripper joint exhibited the highest error (0.3451 radians), suggesting the need for improvements in its control mechanism. The trained model's predictions were tested in a simulated environment, where angles are sent to Gazebo via ROS Actions. These results demonstrate the potential of the proposed approach while highlighting opportunities to enhance the gripper's control by simplifying its operation through distance-based mechanisms or by refining data collection methods.