Detection of Deepfake Texts Generated by Large Language Models: An Empirical Audit and Detection Framework for the Turkish Language

Detectors of text generated by large language models can achieve high scores by exploiting shared source documents, data sources, or generator-specific artifacts rather than genuine linguistic signals. This is a serious concern for textual deepfakes written to mislead readers or impersonate human authors. We present a methodological audit of a Turkish human-versus-synthetic text classifier. The archived BERTurk model reproduced its near-perfect row-random test result (F1 = 0.9990), but 86.38% of test samples shared a root document with the training data. We then built a corrected 215,074-sample cohort from a single generator (Qwen2.5-7B-Instruct) and four development sources, using a group-disjoint split and three seeds. Within this cohort, BERTurk achieved F1 = 0.9958 ± 0.0001 (XLM-R: 0.9923 ± 0.0006), whereas cross-source testing lowered F1 to 0.8990. In a separate, prospectively frozen extension, nine new BERTurk models evaluated on held-out generator families reached a mean macro-F1 of 0.9379. However, on 500 independent Turkish Wikipedia texts, the same nine models showed a mean false-positive rate of 0.7384 and a balanced macro-F1 of 0.5500, failing all external-validity criteria. High in-cohort scores therefore do not indicate general performance beyond the evaluated generators and sources. The main contribution is a traceable evaluation protocol that separates historical, corrected, and failed-validity results.