An Analysis of Current Situation and Prospect of Natural Language Processing Application in Discourse Analysis

Abstract. Natural Language Processing is a subfield of Artificial Intelligence and an interdisciplinary field of Linguistics and Computer Science. Previous research mainly focuses on the comparison and practical application of natural language processing technology, but the amount of research on its applications in Linguistics is limited. To fill this gap, this article focuses on the current application of technologies in discourse analysis, which is one of the most popular fields in Linguistics. The researcher will initially summarize the existing techniques and tools of Natural Language Processing used for pre-processing, such as Tokenization, Stemming, and Named Entity Recognition. Open-source tools such as NLTK and Spacy for pre-processing are also mentioned in the article. Subsequently, the researcher will introduce commonly used technologies like sentiment analysis and semantic analysis. This research concludes that discourse analysis can benefit many industries such as Medicine, Economics, Agriculture and Politics. However, this paper also analyzes the drawbacks and challenges that NLP technologies for discourse analysis face. Barriers such as Word Sense Disambiguation (WSD) and the context dependency of human language made it difficult for NLP technologies to be applied in Pragmatic analysis.