Research on AI Translation Based on Natural Language Processing

In this paper, a sample set with good performance will be selected from a large number of existing parallel corpora and used for statistical machine translation. The method of word repetition extraction is used to analyze the English and Chinese patent texts. The method based on block analysis is used to filter the phrase repeated extraction. In order to solve the problems such as alignment error and semantic ambiguity in phrase repetition extraction, this project adopts the method of rearranging phrase repetition extraction based on distributed similarity. This project will consider the extent of sentence coverage and the impact of sentences on translation, and study how to extract high-quality small sample data sets from existing data. The test of CWMT2008 between Chinese and English proves that the algorithm can effectively select the translation corpus with high level from the massive data. After compressing 80% of the training samples, the translation quality similar to Baseline algorithm can be obtained. In addition, the reclassification method based on distributed similarity can improve the extraction efficiency.