Classification of the Noisy Texts Based on Feature Vectors

Quantitative Linguistics (QL) employs mathematical models to analyze linguistic systems, revealing interrelations expressible only through numbers or rankings. These universal, stochastic laws apply across all linguistic levels and languages. Text corruption analysis is crucial for ensuring data integrity, error detection, and communication reliability, while addressing security, protocol efficiency, data consistency, and user trust. This study proposes a method to enhance service levels in unreliable message transmission applications through text corruption detection. Analysis of 20 CIA texts revealed that feature vectors derived from part-of-speech tagging exhibit specific patterns independent of text size. Subtexts showed reduced cohesion with parent texts as their size decreased. An exponential function fitted the mean intensity pattern of sorted token identifiers, indicating a common linguistic structure in military and political texts. Noise introduction experiments showed higher noise levels reduced the mean coefficient of variation and increased entropy fluctuations, with low noise levels having minimal impact. These findings suggest that monitoring coefficient of variation and entropy metrics can preserve text integrity, even under noisy conditions, providing a quantitative framework for understanding text corruption effects and supporting the development of reliable communication systems and linguistic models.