RIAND: Robustness-Improved and Accelerated Neural-Deduplication

The predominant approach to text deduplication is still lexical-based, failing short in detecting near-duplicates that have low lexical similarity but high semantic similarity. Neural deduplication methods address those issues using embed dings to capture surface text and semantic meaning. However, they face challenges such as sensitivity to noise, limited scalability, and lacked effectiveness in real-world corpora. This study aims to overcome these weaknesses. We enhance noise robustness through noised data augmentation and contrastive learning, ensuring similar embed dings for sentences and their noisy counterparts. We also improve scalability with lower theoretical complexity, an optimized running pipeline, and distributed processing, resulting in a significant speed improvement. Applying our method to real-world corpora demonstrates its effectiveness, and we observe performance gains in language models trained on deduplicated data. Qualitative analysis confirms the superiority of the RIAND method in identifying near-duplicates.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>https://githuh.com/PosoSAgapo/TextDupRIAND