Beyond Lexical Conformity: Comparing Automatic Indicators and Expert Judgments in TOCFL-Controlled Chinese Text Generation

Large language models (LLMs) generate fluent Chinese, but controlling lexical difficulty remains challenging. This study examined a Python-orchestrated external-control workflow using one GPT model, two sequential prompting conditions (no-word-list and lexicon-aware), three post-generation revision strategies (word-level substitution, sentence-level rewriting, and downstream regeneration), and four Test of Chinese as a Foreign Language (TOCFL) levels (A1–B2). Automatic indicators covered 96 initial drafts and 288 revision outputs. For 24 selected tasks, two experts directly rated 180 text instances; 12 unchanged outputs inherited initial-draft ratings, yielding 192 condition-specific expert-evaluation records. Task-matched comparisons used 20,000 TOCFL-stratified task-cluster bootstrap resamples and Holm-adjusted tests; automatic–expert associations were analyzed using Spearman correlations with task-cluster confidence intervals. Lexicon-aware prompting was associated with fewer initial out-of-level words and less revision, although the prompting batches were sequential. Word-level substitution and sentence-level rewriting generally showed greater source-relative embedding resemblance, whereas downstream regeneration received higher expert ratings for several language-quality dimensions but not consistently for level appropriateness. Residual out-of-level word density showed a weak, inconclusive pooled association with level appropriateness. Whole-text cosine similarity was negatively associated with semantic plausibility overall but positively associated within word-level substitution. Automatic indicators and source-blind expert judgments thus provide complementary, non-interchangeable evidence.