Enhancing Language Model Performance with a Novel Text Preprocessing Method
Advances in natural language processing highlight the importance of text data preparation with machine learning. It has been reported that the traditional methods often fail to deal with the language complexity which affects model performance. Consequently, this paper proposes an approach which uses tokenization, noise reduction, and normalization to improve text quality.
Authors
- Annamária R. Várkonyi-Kóczy (ORCID: https://orcid.org/0000-0002-6932-8608)
- Hamid Tabrizchi
- Amir Mosavi
- Amirreza Jalili (ORCID: https://orcid.org/0009-0008-5051-1067)
Institutions
- Obuda University (HU)
- Ludovika University of Public Service (HU)
- J. Selye University (SK)
- University of Tehran (IR)
- University of Tabriz (IR)
Publication Details
- Journal
- Acta Physica Polonica A
- Published
- 2024-10-01
- DOI
- https://doi.org/10.12693/aphyspola.146.542
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00