Cross-domain aspect term extraction using pre-trained language models with pre-training and fine-tuning strategy

As an important subtask of fine-grained sentiment analysis (SA), aspect term extraction (ATE) aims to identify aspect terms within user-generated comments. ATE-supervised learning approaches are based heavily on the availability of annotated data with token-level labels. However, obtaining these annotations for each domain in sufficient quantities is often a costly process, which limits the applicability of supervised methods. The cross-domain aspect term extraction (CDATE) problem involves transferring knowledge from a well-annotated source domain to a sparsely annotated target domain. We propose an approach that combines a pre-trained language model with a pre-training and fine-tuning strategy for solving CDATE (named LM-PF). The first stage adapts BERT using unlabeled data from both domains to capture do-main-specific characteristics, the second stage fine-tunes a downstream sequence labeling model with minimal labeled data to improve target domain performance, the third stage pre-trains on source-domain data and fine-tunes the bidirectional long short-term memory with conditional random fields (Bi-LSTM+CRF) with limited target labels. Specifically, we first adapted a pre-trained BERT model using unlabeled data from both the source and target domains to better capture domain-specific language patterns. Then, we built an ATE model initialized with this adapted BERT and pre-trained it using labeled source domain data. Finally, we fine-tuned the model on the labeled target domain data to enhance its generalization to new domains. We conducted six CDATE tasks using three benchmark datasets, restaurants, laptops, and digital devices. The results show that our model achieves the highest average Micro-F1 score (60.09%) across all tasks and outperforms strong baselines such as generative cross-domain data augmentation (GCDDA) by an average margin of 2.7%, confirming the effectiveness of combining domain-adaptive pre-training with task-specific fine-tuning in improving cross-domain generalization for ATE tasks.