Text Substitution in Natural Language Processing (NLP)

This article explores the often-overlooked yet critical step of text substitution in natural language processing (NLP). Covering contractions, HTML/XML cleaning, unit standardisation, and domain-specific preprocessing, it demonstrates how substitution enhances model efficiency and semantic clarity. Through practical code, it shows how consistent representation improves tokenisation and machine interpretation. The session bridges theory and practice, offering both foundational principles and research-oriented challenges in creating customised pipelines for varied NLP tasks, from general corpus cleaning to physics education systems like AINSTEIN.