Fine-Tuning Transformers for News Topic Classification: Method and Best Practice

Topic classification is often only a trivial thing in many people's views. But in fact, topic classification can improve user experience, especially for users of digital platforms. Topic classification of news articles enables personalised news feeds, optimises information retrieval, and facilitates content management. Efforts to develop news topic classification have been made with various traditional and deep learning techniques, but there are still challenges to overcome, such as the need to improve classification accuracy and minimise computational costs. In this study, these challenges were addressed by fine-tuning several pre-trained models, including BERT, RoBERTa, XLNet, ALBERT, and DistilBERT, on the BBC Full Text Document Classification dataset. Various hyperparameter settings, focusing on learning rate and batch size, were explored to identify the optimal configuration. The results show that while RoBERTa achieves the highest accuracy and F1 score, DistilBERT, with a learning rate of 1e-5 and a batch size of 64, offers a competitive alternative by achieving an accuracy of 98.2% with the shortest average training time. DistilBERT outperforms all models in training time, indicating that high performance can be achieved with lower computational demands. This highlights the potential of DistilBERT as a more efficient model for news topic classification, balancing performance with computational efficiency.