Japanese Tokenization using Simple Associative Arrays

This study addresses a technical challenge in text analysis for the practice of Institutional Research (IR). In the research field of Natural Language Processing (NLP), development focuses on accuracy and speed. Therefore, suitable data for evaluation and efficient programming languages are used. On the other hand, tasks outside the scope of NLP are also important in IR. For example, not only textual data, but also numerical data must be analyzed, and in some cases, the analysis of data stored in web servers is required. Since NLP and IR involve different tasks, the tools required are also different. One serious problem for text analysis in IR tasks is the limited development of Japanese tokenization. In this study, we implement a Japanese tokenizer, focusing on its applicability in web servers. Since function libraries in scripting languages for web environments are limited, our main technical challenge is to achieve efficiency in tokenization by combining the available functions. To deal with this problem, our method uses associative arrays to reduce unnecessary steps in tokenization. Evaluation experiments were conducted using Japanese text data, and the results showed that our tokenizer is viable in web server environments.