Can large language models generate novel scientific ideas? A comprehensive study on data-driven astronomy

Scientific discovery is a cornerstone of societal advancement, and its rapid development demands innovative tools that can facilitate the generation of research ideas. Recent breakthroughs in Generative Artificial Intelligence (GenAI), particularly in Large Language Models (LLMs), offer transformative potential for scientific idea generation. However, existing LLM-based idea generation methods are limited to computer science and closely related domains, and their application in other scientific fields, e.g., astronomy, remains largely underexplored. In this paper, we investigate the applicability of LLMs for scientific idea generation in data-driven astronomy, an interdisciplinary field that applies advanced data science methods to analyze massive datasets collected by modern telescopes and satellites to drive astronomical discoveries. Specifically, we implement a novel framework, AstroInsight, that integrates conception, iterative refinement, expert validation, and knowledge integration for idea generation. Through extensive experiments with human expert assessments and model self-evaluation, we show that AstroInsight effectively generates new research concepts and substantially accelerates discovery cycles, with the generated drafts achieving a novelty score of 3+/6 in both human- and model-based evaluations. Additionally, its generated ideas match or exceed human-generated ones in terms of originality and feasibility across multiple topics. In summary, we provide researchers with tools to boost productivity while maintaining rigor in a human-AI collaborative framework, thereby illuminating pathways to building LLM-assisted systems for autonomous scientific ideation.