Liar, Liar, LLM on Fire: Investigating Deception in Turkish Text Generation

“Is a Large Language Model (LLM) a good liar? Can we detect its lies for Turkish?” LLMs are capable of generating deceptive or misleading information, which can be considered as “lying”. In this paper, we introduce a dataset called TQuADFake , consisting of correct and incorrect sentences generated by LLMs through the Question Answering (QA) task. We evaluate the quality of the dataset using both human and LLM evaluators. We explore linguistic features that may help distinguish incorrect sentences and build a lie detection model based on these features. Our analysis shows that while linguistic features are useful for identifying differences between correct and incorrect sentences, these features alone are insufficient to fully solve the lie detection problem. We also investigate whether LLMs are aware of their own inaccuracies and compare various models’ performance on this task, including SVM, pre-trained language models (PLMs), and LLMs. This study provides a foundation for future research on misinformation detection in Turkish and highlights the challenges in building effective lie detection systems.