Enhanced Semantic Relatedness Prediction Using XLM‐RoBERTa and CNNs With K‐Fold Cross‐Validation

ABSTRACT Accurately measuring the semantic relatedness between sentences is crucial for various natural language processing (NLP) tasks, including question answering, text summarization, and information retrieval. This study introduces a system designed to precisely evaluate the relatedness of English sentences. Utilizing the SemRel2024 dataset, a comprehensive benchmark for semantic textual relatedness (STR), we conducted baseline experiments across multiple monolingual settings. Our proposed method integrates k‐fold cross‐validation with a fine‐tuned XLMRoBERTa‐large model and a convolutional neural network (CNN) architecture, achieving the highest Spearman's correlation of 0.854 and Pearson's correlation of 0.863. We also explored several transformer‐based models (RoBERTa‐base, RoBERTa‐large, BERT‐large) and their architectural variations, as well as the effects of attention mechanisms with word embeddings such as Word2Vec, FastText, and global vectors (GloVe). Among traditional machine learning models, the TF‐IDF + random forest (RF) model exhibited the best performance. Our findings demonstrate the potential for significant advancements in NLP applications through enhanced semantic relatedness prediction, thereby improving machine translation, information retrieval, and the development of sophisticated language models capable of nuanced understanding.