Comparative evaluation of similarity metrics for Chinese retrieval augmented generation

Large language models (LLMs) have achieved impressive performance in natural language generation but remain vulnerable to hallucinations, particularly in Chinese short-text tasks where evidence grounding is essential. Retrieval-Augmented Generation (RAG) offers a promising solution, yet most prior studies rely on cosine similarity over a single embedding model and focus primarily on English benchmarks. In this study, we systematically evaluate eight similarity metrics—Jaccard, Weighted Jaccard, Dice, character n-gram Jaccard, Sentence-BERT cosine, Euclidean, Manhattan, and Levenshtein—together with two strong Chinese text embeddings (bge-base-zh-v1.5 and m3e-base) within a unified Chinese RAG pipeline. Using the complete CMRC2018 development set (3,219 questions), a second Chinese dataset (DuReader-robust, 1,417 questions), and two deterministically decoded generators (DeepSeek-R1-Distill-Qwen-14B and Qwen2.5-7B-Instruct), we report paired significance tests (McNemar exact; paired bootstrap) with Holm–Bonferroni correction and effect sizes. Four findings emerge. First, strong Chinese embeddings dominate retrieval on both datasets (bge: Recall@4 = 95.3% on CMRC2018, 98.3% on DuReader-robust), showing that the weak retrieval of the multilingual MiniLM SBERT encoder (62.5% on CMRC2018, 68.5% on DuReader-robust) is a property of that encoder rather than of embeddings per se. Second, once a length-bias implementation artifact is corrected, windowed character-level edit similarity attains Recall@4 = 93.8% and end-task F1/EM/grounding statistically indistinguishable from bge—without any embedding infrastructure. This advantage, however, is dataset-conditional: on DuReader-robust’s short keyword-style queries (mean 9.4 characters), which share few verbatim anchors with their source passages, the same lexical metrics fall to 55.8–67.6% and no longer beat even the multilingual SBERT baseline—bounding the scope of this finding to anchor-rich extractive QA. Third, resampling from full-set results shows that evaluations with n ≤ 100 select the wrong best method in up to 37.6% of draws, quantifying the risk of small-sample RAG studies. Fourth, method rankings are stable across the two generators at the tier level (Spearman ρ = 0.70–0.90) while absolute scores are strongly generator-dependent, and a single-threaded computational profile shows that competitive lexical retrieval needs no model weights at 17–22 ms per query, versus a 409 MB encoder and an 8–11 ms per-query encoding pass for the best embedding. For the datasets, encoders, and generators tested, these results provide statistically grounded, hardware-realistic guidance for building efficient and reliable Chinese RAG systems.