Hyperparameter Tuning to Improve Object Detection Performance in Handwritten Images

Object detection in handwritten images is a complex challenge due to variations in writing styles, letter sizes, and environmental conditions when handwriting is created. The You Only Look Once (YOLO) method has shown promising results in detecting objects in real-time in various types of images. However, YOLO performance is highly dependent on optimal hyperparameter tuning for a specific detection task. Hyperparameter tuning is a crucial step in improving object detection performance, especially in the context of handwritten images. This research aims to optimize YOLO hyperparameters to increase the accuracy of object detection in handwritten images. Some tuned hyperparameters are momentum, weight_decay, learning rate, batch size, and epoch. This research was carried out for 49 training sessions using the YOLOv8 and CUDA version 11.8 frameworks with a primary dataset obtained of 44 pieces of diverse handwriting, including various variations of handwriting styles that were consistent in lighting conditions, type and size of paper and pen, form format, and healthy physical and psychological conditions. The results of this research show that hyperparameter tuning in experiment 34 with parameter lr0 of 6.879476049637327e-05, momentum of 0.963981540467, weight_decay of 0.00011207385, batch of 2, and imgsz of 480, obtained the model performance evaluation metric, namely F1-score is 35%, Recall is 52%, Precision-Recall is 36%, and MAP is 0.5. The temporary results of this research are still not good so there are still some improvements needed, such as annotating handwritten images with greater variations by using polygon types as bounding, changing parameters such as the number of epochs, and increasing the number of datasets. This research is useful for things like recognizing students' personalities, forensic identification, and also patient therapy using handwriting.