Knowledge distillation of attention-refined transformers for aspect-based sentiment analysis

Aspect-Based Sentiment Analysis (ABSA) is a fine-grained natural language processing task that identifies sentiment polarity toward specific aspects within a sentence. Pre-trained language models such as BERT are widely used for ABSA. However, most current research captures semantic relations between target and context words in an implicit manner, yet often lacks explicit aspect–context interaction modeling. In addition, considerable computational costs, large memory footprints, and high inference latency pose critical challenges to the deployment of these models on resource-constrained edge devices. To address these limitations, we propose SMART (Sentiment Model with Attention-Refined Transformers), a two-stage framework whose teacher couples a shared BiLSTM, an Attention-Over-Attention module for explicit aspect-context alignment, and a partially frozen BERT encoder, and which is then distilled into two lightweight students: one retaining the explicit aspect module without pre-training (AOA), and one retaining pre-training without an aspect module (DistilBERT-SPC). Experiments on five datasets, five random seeds, and two distillation objectives, three findings emerge. First, explicit aspect attention is measurable only where aspect-level and sentence-level sentiment diverge: ablation changes performance by less than seed noise on aggregate benchmarks, yet the teacher attains its largest margin over every competitor, including an 8-billion-parameter language model under in-context learning, on the sole multi-aspect corpus. Second, response-based distillation transfers class knowledge in proportion to class frequency, so minority-class discrimination degrades systematically; reweighting the supervision term reallocates precision against recall without closing the gap, indicating that imbalance-aware distillation must act on the transfer term instead. Third, pre-trained representation dominates structural alignment with the teacher: the two students are statistically indistinguishable given sufficient data but diverge sharply below roughly one thousand training examples. The distilled students reduce memory by 99.8 % and run 6.6 \\(\\times\\) faster than the teacher, establishing feasibility for real-time deployment.