Fast Retrieval Algorithm of Multilingual Translation Corpus Based on Distributed Storage

With the acceleration of globalization, massive multilingual translation corpora in distributed storage environments present core problems such as strong data heterogeneity and low cross-node retrieval efficiency. To this end, this study proposes a cross-node retrieval optimization framework based on a hybrid index structure. Firstly, this paper designs a multi-dimensional feature coder for semantic units. Then, this paper constructs a distributed hierarchical index structure based on improved LSH (Locality-Sensitive Hashing). Finally, a dynamic load balancing algorithm based on spark is developed. Experimental results show that on the Europarl and UN Parallel 2.1 TB multilingual corpus, the Top-10 retrieval accuracy is as high as 91%, and the Hybrid scheduling strategy achieves 800 QPS throughput under 1500 concurrent conditions. In the above data conclusions, this framework performs well in retrieval accuracy, real-time responsiveness and high concurrent throughput, providing scalable technical support for real-time retrieval of cross-border language service systems.