(153) CAN LARGE-LANGUAGE MODELS ACCURATELY DISCERN AI-GENERATED SEXUAL MEDICINE SCIENTIFIC LITERATURE FROM HUMAN GENERATED?

Abstract Introduction Chat-GPT an iteration of Artificial Intelligence (AI) which utilizes Large-Language Model (LLM) to understand and generate text, has been increasingly employed in scientific literature, healthcare, and education since its release in 2022. While these LLMs offer the possibility of increasing the clarity of scientific writing, they have been met with criticism regarding utility, accuracy, legality, and reliability. To combat potential abuse of these powerful tools, academic institutions have used AI tools to help identify text generated by an LLM. However, no study has currently validated the accuracy and utility of AI in this setting. Objective To validate if commercially available AI detector tools can identify potentially AI generated text within published sexual medicine articles. Secondary aims included identifying factors that contribute to an article being flagged as AI generated. Methods Phrasly.AI was utilized as the primary AI detector tool that cross-referenced scores from other AI detectors named “ContentAtScale”, “GPTZero”, “ZeroGPT”, “OpenAI”, “CopyLeaks”, “TurnItIn”. Top 20 most cited articles published in the Journal of Sexual Medicine were obtained for the following 5 year intervals: 2004-2008, 2009-2013, 2014-2018, 2019-2024. Positive controls of known AI generated articles were also obtained. Each of these articles and controls were analyzed with Phrasly.AI. The following variables were obtained: type of article, sentences analyzed, number of sentences likely to be AI generated, percent of article AI-generated, country of authorship, and a likelihood AI-generated score (1 = Human, 2 = partial AI, 3 = complete AI). Results were analyzed using program R utilizing binominal regression models to determine likelihood of obtaining an AI-Generated score. Results A total of 80 articles in total were analyzed with 5 positive controls. All positive control articles were predicted to be AI-generated. 45% (36/80) articles were flagged as likely AI-Generated. All articles analyzed were published prior to 2022 when the first LLM was released. No significant association was found between the likelihood AI-Generated score and the following variables: country of authorship and the year published of the article. We did observe a significant association between type of article and likelihood of being AI-Generated in our regression model showing review articles had an OR 3.10 (95% 1.19-8.55; p = 0.023). Conclusions Current AI detector tools often inaccurately discern human-generated scientific writing as AI-Generated as all articles included in this study were published prior to the releast of Chat-GPT. Review Articles are three times as likely to be flagged as AI-generated. Further validity of the utility and accuracy of these tools is needed prior to their widespread adoption in cross-checking for AI utilization in the construction of scientific literature. Disclosure No.