Construction and Alignment Features of English Corpus based on Particle Swarm Optimization Algorithm
With the continuous advancement of natural language processing technology, building a high-quality corpus has become a hot topic in current research. However, the traditional corpus construction method requires a lot of manual intervention, which is time-consuming and laborious, and cannot meet the requirements of large amounts of data. To overcome these limitations, this paper applies an automated method based on the particle swarm optimization (PSO) algorithm. First, the scale and text type of the corpus are determined, and then an adaptive function suitable for text similarity evaluation is designed and evaluated. The method mainly consists of three parts: initialization of the particle swarm, updating of particle positions, and selection of the best solution. Through experimental evaluation, we found that in a large sample scenario, the perplexity of the PSO algorithm can be stabilized at around 2, showing good robustness. In addition, the evaluation using the BLEU (bilingual evaluation) score also reached a high score of 0.99, which indicates the high quality of the alignment. The research results will further verify the feasibility of the PSO algorithm in the automatic construction and alignment of the English corpus, and provide a new and efficient method for corpus construction in natural language processing. Through automated methods, manual participation can be significantly reduced, construction efficiency can be improved, and alignment quality can be maintained or improved.
Authors
- Xin Li (ORCID: https://orcid.org/0000-0002-5232-7465)
- Wei Shang (ORCID: https://orcid.org/0000-0002-6788-4969)
Publication Details
- Published
- 2024-09-20
- DOI
- https://doi.org/10.1109/icdscnc62492.2024.10939378
- Primary Topic
- Educational Technology and Pedagogy
- Type
- article
- Field-Weighted Citation Impact
- 0.00