Dual Embedding and Fusion-Based Architecture for Multi-Label Classification in Customer Support Systems

This paper introduces an innovative dual embedding and fusion-based architecture for multi-label classification in customer support systems, addressing the critical challenge of automating IT support ticket triage. Modern organizations face mounting inefficiencies with traditional manual triage methods, resulting in delayed response times, inconsistent handling, and diminished user satisfaction. Our proposed architecture uniquely leverages the complementary strengths of two powerful embedding models—BERT and Ollama—to create a robust classification system. While BERT excels at capturing bidirectional context and technical terminology through its 768-dimensional contextualized token embeddings, Ollama-based embeddings demonstrate superior capabilities in understanding nuanced language patterns and implicit contextual cues, particularly for ambiguous or unconventional phrasings. Our methodology consists of five key components: (1) comprehensive data preprocessing to ensure standardized, clean input; (2) parallel embedding generation through both BERT and Ollama paths; (3) dimension harmonization that projects both embeddings into a common 512-dimensional space using carefully designed projection networks with ReLU activation, batch normalization, and residual connections; (4) sophisticated multi-head attention fusion with 8 attention heads where BERT-derived embeddings function as queries while Ollama-derived embeddings serve as keys and values; and (5) feature extraction that systematically distills representations to 128 dimensions before final classification through a multi-layer architecture with strategic dropout for regularization. Experiments conducted on a dataset of 20,000 anonymized multilingual customer support tickets across five languages and diverse business domains demonstrate the superior performance of our proposed model (86.70% accuracy, 87.25% precision, 86.83% recall, and 87.04% F1 score) compared to traditional machine learning approaches like Logistic Regression (65.79% accuracy), Random Forest, and XGBoost (74.32% accuracy), as well as deep learning models including LSTM and BiLSTM (82.85% accuracy). The confusion matrix analysis further validates the model's effectiveness across ten service categories, with particularly strong performance in technical support, product support, and customer service classifications, while maintaining reasonable precision even for categories with fewer samples. This architecture represents a significant advancement in automated ticket triage, offering potential for substantial improvements in operational efficiency, resource allocation, and service quality in complex IT support environments.