Feature guided prompting distorts synthetic fake news corpora without improving detection

Large language models (LLMs) have been increasingly applied for generating synthetic training data to address class imbalance problems in fake news detection. Existing approaches commonly apply multiple stylistic constraints in generation prompts to control the characteristics of generated samples and evaluate their overall compliance. However, the contribution of individual constraints to synthetic data augmentation performance remains unclear. This research investigates whether any single stylistic constraint is necessary or sufficient to improve LLM-based data augmentation effectiveness. A single-feature ablation analysis was conducted using an LLM-based augmentation pipeline with the GossipCop–PolitiFact headline dataset containing 23,196 samples and 15 stylistic constraints adopted from previous research. The Qwen2.5-7B-Instruct model was used as the primary generator across 26 prompt conditions. Phi-3.5-mini-instruct and DeepSeek-R1-Distill-Llama-8B were used for cross-model validation on three key conditions (full, naive, F5-only). Three independently generated corpora, each containing 5,000 headlines, were produced for every condition and evaluated with six classical classifiers (30 seeds each), three fine-tuned transformers (3 seeds each), and one zero-shot LLM classifier. The results indicate that no individual stylistic constraint is necessary, as all 15 ablation settings achieved statistical equivalence under a two-one-sided equivalence test with δ = 0.02. Although three features (F1, F7, F9) produced statistically detectable changes after FDR (false discovery rate) correction, their magnitudes remained within the equivalence margin. Furthermore, the F5-only prompt achieved performance within δ = 0.02 of the full prompt across the three evaluated generator families. The maximum within-family F1 difference among the full, naive, and F5-only conditions was 0.0088. Although F5-only achieved numerically higher F1 scores than naive prompting, the differences remained below the predefined practical margin. In the Qwen experiments, the F6-only and F12-only prompts also matched the full prompt within δ = 0.02, as did the F6 + F4 condition (Δ = −0.0037). In the primary Random Forest evaluation, the Qwen full, naive, and F5-only conditions achieved lower fake F1 scores than the no-augmentation baseline of 0.6034. Random oversampling and class-weighted Random Forest achieved higher scores of 0.6324 and 0.6452, respectively. The findings indicate that the stylistic constraints may introduce distributional distortions rather than improve alignment with real fake-news characteristics. The full prompt increased question mark frequency by 5.3× and reduced ALL CAPS usage by 12.5×, while the F5-only prompt increased exclamation mark frequency by 15×. The direction of question-mark distortion is partly model-specific: Qwen and Phi over-comply (5.34× and 4.51×) while DeepSeek under-complies (0.66×), confirming that the misalignment is a joint property of the prompt and the generator. These findings indicate that synthetic data quality evaluation should consider feature-level distributional alignment rather than relying solely on aggregate prompt compliance, particularly within the scope of the English-language headline generation setting evaluated in this study.