Hierarchical Text Classification: Fine-tuned GPT-2 vs BERT-BiLSTM

Abstract Hierarchical Text Classification (HTC) is a specialised task in natural language processing that involves categorising text into a hierarchical structure of classes. This approach is particularly valuable in several domains, such as document organisation, sentiment analysis, and information retrieval, where classification schemas naturally form hierarchical structures. In this paper, we propose and compare two deep learning-based models for HTC. The first model involves fine-tuning GPT-2, a large language model (LLM), specifically for hierarchical classification tasks. Fine-tuning adapts GPT-2’s extensive pre-trained knowledge to the nuances of hierarchical classification. The second model leverages BERT for text preprocessing and encoding, followed by a BiLSTM layer for the classification process. Experimental results demonstrate that the fine-tuned GPT-2 model significantly outperforms the BERT-BiLSTM model in accuracy and F1 scores, underscoring the advantages of using advanced LLMs for hierarchical text classification.