Unraveling Paraphrase Detection: A Deep Dive into Transformer Models using the MRPC dataset Before LLMs

Paraphrase identification remains a key Natural Language Processing (NLP) task, with the Microsoft Research Paraphrase Corpus (MRPC) dataset widely used for evaluating model performance. This paper presents an experiment to evaluate the effectiveness of transformer-based models, specifically BERT, RoBERTa, and DistilBERT, for paraphrase identification, without the usage of Large Language Models (LLMs), which have become dominant in recent years. The paper focuses on performance with Inductive Conformal Prediction (ICP) and XCP (3-fold cross validation) methods, with an emphasis on evaluating Macro F1 scores and average set sizes for each model. Our findings show that RoBERTa-base outperforms BERT-base with a Macro F1 score of 0.8476 compared to 0.7366, indicating that transformer models before LLMS are still giving valuable outcomes for paraphrase detection. DistilBERT-base achieved a Macro F1 of 0.7871, demonstrating its effectiveness, despite being a lighter model. The study explores the challenges of such models, particularly in terms of capturing structural differences and lexical overlap in paraphrase identification tasks. We also touch on the importance of using pre-LLM transformers in the modern NLP landscape, highlighting the key points of the approach. This research aims to shed light on the performance of transformer models before the rise of LLMs, providing insights on how the models still have the potential to be beneficial f or paraphrase identification without the advanced capabilities of present LLMs.