To Combine or Not to Combine? The Influence of Combining Training Datasets on the Robustness of Deep Learning Models: An Analysis for Optical Character Recognition of Handwriting

The present manuscript addresses the question of how training data should be sampled for deep learning models by analyzing and evaluating the impact of training data representation and complexity on the performance and robustness of deep learning models. To address this open question, we take a combinatorial approach and train three architecturally identical deep learning models on three combinations of handwritten digit datasets of varying complexity: EMNIST Digits, DIDA, and a newly composed third dataset combining the first two datasets. Each model was evaluated using withheld test data from all three datasets. We find that models trained exclusively on either EMNIST Digits or DIDA performed well on their respective datasets but poorly on unfamiliar datasets. However, the model trained on both datasets showed an overall solid performance, although not quite reaching the accuracy of the specialized models on their respective datasets. We conclude that while specializing in the training dataset can increase accuracy, a more diverse dataset enhances model robustness. In practice, deep learning models should thus be trained on data that represents the actual application environment as closely as possible or, if such data is not available, on diverse data.