Scalable Vocabulary Pruning for Large-Scale Chinese Word Segmentation on GDELT Ngram 3.0

This study addresses the challenge of Chinese word segmentation (CWS) in the only open, large-scale, real-time multilingual news dataset: GDELT Ngram 3.0. Conventional CWS systems are not designed for the structured and redundant format used in GDELT Ngram data and face limitations when scaled to this setting. To address this, a PySpark-based segmentation approach is proposed and evaluated across multiple data formats, prioritizing processing efficiency while maintaining acceptable segmentation quality. This approach incorporates scalable algorithms for the distributed computation of word boundary scores, utilizing frequency- and association-based statistical measures. Experiments demonstrate that the PySpark-based approach outperforms existing efficiency–performance trade-offs on GDELT Ngram-formatted evaluation data synthesized from standard CWS test-sets, effectively establishing a new Pareto frontier. It achieves an F1 score of 52%, with 64% recall and 44% precision, while delivering nearly twofold speedup over the fastest conventional system. Trade-off analysis on a development set further reveals that, at approximately 95% recall, the proposed pruning method can reduce the empirical candidate space by 70% and the vocabulary size by 99.85%. These findings validate the feasibility of real-time, distributed CWS for massive streaming news datasets and lay the groundwork for extending GDELT Ngram 3.0 with segmented word Ngram data for scriptio continua languages in the future.