Leveraging Transformers and Transfer Learning for Enhanced Document Classification

This research systematically explores transformerbased models enhanced by transfer learning for document classification, addressing limitations of traditional methods like SVMs and Naive Bayes that struggle with feature engineering and language complexity. While deep learning approaches using CNNs/RNNs improved performance, their inability to capture long-range dependencies was overcome by transformer architectures employing self-attention mechanisms. The study demonstrates that multimodal pretrained models—notably LayoutLMv3 and DocFormer—significantly outperform textonly transformers (e.g., BERT variants) by integrating OCRextracted text with visual layout features through 2D positional embeddings and vision encoders, achieving $\sim 90 \%$ accuracy/F1score on datasets like RVL-CDIP. Critical to this success are training methodologies including. However, these gains come with computational trade-offs: multimodal models require substantially longer training times ($\mathbf{3 6 - 3 8}$ hours) compared to textbased alternatives (BERT: 18 hours), while efficiency-focused variants like ALBERT sacrifice accuracy for speed. Future work must address optimization challenges, interpretability limitations, and expansion to diverse document types like handwritten manuscripts to enable broader industrial adoption.