Can LLMs Beat Humans on Discerning Human-written and LLM-generated Science News?

Science news is increasingly important in connecting scientists and the public by sharing discoveries and innovations. With the rise of large language models (LLMs), there is potential to automate science news creation, but concerns exist about the quality of LLM-generated news versus human-written news. This paper explores whether LLMs can outperform humans in distinguishing between human-written and LLM-generated news. Inspired by the Chain-of-Thought prompting method, we designed a simple yet effective variant called Guided Few-shot (GFS), which encodes the characteristics of news of two types with examples. Our experiments indicated that GFS with just a single example effectively boosted the performance of all LLMs and made open-weight LLMs achieve or exceed the performance of humans and commercial LLMs. The code and data are available in our repository at https://github.com/lamps-lab/sanews.