Transformers for Domain-Specific Text Classification: A Case Study in the Banking Sector

The growing volume of unstructured text data in the banking sector has created a need for advanced classification methods to manage customer inquiries efficiently, resulting in faster response times, automated message classification, and reduced human errors. The classification results are integrated into live banking systems, enabling continuous 24/7 message processing and instant categorization. Previously, human operators could process up to 70 messages per day during working hours. With AI integration, the system now categorizes messages automatically and in real time, ensuring that relevant department managers receive them promptly, significantly improving operational workflows. Our research offers key insights into applying transformer models for domain-specific classification in banking. This paper presents a case study on fine-tuning transformer models—BERT, GPT-2, GPT-3, and Falcon-1B for domain-specific text classification in the banking industry. The dataset used in this investigation consists of 5,447 customer messages submitted between 2023 and 2024 through Invest Bankś secure messaging portal, which serves as a direct communication channel where customers can submit their requests at any time of the day without needing to visit the bank in person. The inquiries span fifteen service categories provided by the bank. To the best of our knowledge, no prior studies have focused on automating customer secure messages in the banking sector, especially those written in colloquial and abbreviated language. The models were fine-tuned to improve classification accuracy and operational efficiency. A heterogeneous ensemble of BERT and GPT-2 achieved the best performance with an Area Under the Curve (AUC) score of 99.42% and an F1 score of 86.74%. A homogeneous ensemble of BERT models also performed well, achieving an AUC score of 98.81% and an F1 score of 84.57%. Notably, the single BERT and GPT-2 models, with AUC scores of 97.65% and 97.9%, respectively, delivered competitive performance, making them viable alternatives when computational resources are limited.