Dynamic Weight Adjustment in Siamese Multi-Head Self-Attention for Text Similarity

Measuring text similarity is a core task in natural language processing (NLP) with applications spanning across information retrieval, question answering, and paraphrase detection. Despite significant advancements in the field, current models, particularly those based on transformer architectures, often treat all attention heads equally in multi-head self-attention, which may limit the model's ability to focus on the most relevant features for text pair comparison. This paper proposes a novel approach that integrates a dynamic weight adjustment mechanism within a Siamese multi-head self-attention architecture for improved text similarity measurement. The proposed model dynamically adjusts the importance of each attention head based on the input text, enabling the model to emphasize the most informative features of the text. The model is evaluated on the STS-B dataset, where it outperforms several strong baselines, including word2vec, BiLSTM, and fine-tuned BERT models, achieving a Spearman's rank correlation coefficient of 0.889, compared to 0.858 for BERT Base and 0.865 for BERT Large. The results show that dynamic weighting significantly improves the performance of text similarity prediction.