Unified Neural Lexical Analysis Via Two‐Stage Span Tagging

ABSTRACT Lexical analysis is a fundamental task in natural language processing, which involves several subtasks, such as word segmentation (WS), part‐of‐speech (POS) tagging, and named entity recognition (NER). Recent works have shown that taking advantage of relatedness between these subtasks can be beneficial. This paper proposes a unified neural framework to address these subtasks simultaneously. Apart from the sequence tagging paradigm, the proposed method tackles the multitask lexical analysis via two‐stage sequence span classification. Firstly, the model detects the word and named entity boundaries by multi‐label classification over character spans in a sentence. Then, the authors assign POS labels and entity labels for words and named entities by multi‐class classification, respectively. Furthermore, a Gated Task Transformation (GTT) is proposed to encourage the model to share valuable features between tasks. The performance of the proposed model was evaluated on Chinese and Thai public datasets, demonstrating state‐of‐the‐art results.