Integrating textual and visual features to generate pseudo-groundtruth for enhancing training of machine learning models in document image classification

Deep learning-based approaches generally need large labeled datasets for training to show effective performance. However, the manual labeling of datasets is time-consuming and labor-intensive limiting the applicability of deep learning to document analysis. Meanwhile, online digital library portals (e.g., Chronicling America and Trove) offer access to large-scale, often unlabeled document images making them less searchable and discoverable. We propose incorporating conventional textual features from noisy optical character recognition texts and visual features from layouts of document images to generate pseudo-groundtruth for deep learning training. These features train textual base, visual base, and textual–visual base classifiers to predict intermediate pseudo-labels for unlabeled samples. Then, data programming approaches are employed to combine the intermediate pseudo-labels to generate the final pseudo-label. We investigate three data programming approaches—averaging, merge–layer, and Snorkel. The evaluation focuses on the relative performance with and without using pseudo-ground truth under the same condition of limited manually labeled data on two document image classification tasks: poem detection using Aida-17K and multi-class classification using RVL-CDIP. In both tasks, the classifier trained with pseudo-groundtruth had improved performance. The RVL-CDIP classifier presents significantly limited manually labeled training, which achieved ∼55% accuracy with pseudo-groundtruth compared with ∼6% without pseudo-groundtruth using ∼2% of 320,000 training images. Findings indicate that the proposed method demonstrates significant potential to reduce 21 dependency on costly manual labeling, providing a viable solution to effectively train deep learning models for document image classification with limited manually labeled training data. This improvement can further be utilized to enhance accessibility and searchability in digital library portals by efficiently categorizing and segmenting document images.