A Survey of Adaptation of Large Language Models to Idea and Hypothesis Generation: Downstream Task Adaptation, Knowledge Distillation Approaches and Challenges

Idea and hypothesis generation are creative processes that demand a significant level of reasoning. Methods such as brainstorming, analytical reasoning, inductive reasoning and other forms of reasoning have proven useful in advancing research in this domain. Machine learning techniques have been widely investigated to address these challenging tasks. However, they are limited and have insufficient reasoning required for these tasks, making the emergence of language models reignite research in this direction. Large language models (LLMs) have debuted as the current state-of-the-art for achieving impressive generative tasks, and to support language understanding. Models such as the BERT, BARD, GPT and LLaMa have architectural layouts which are mostly transformer network based. These models headline impressive results in downstream tasks such as text classification, sentiment analysis, language inference, question answering, text summarization and named entity recognition among others. However, the need to adapt these models to the emerging downstream tasks of idea and hypothesis generation have uncovered a new research opportunity. In this study, systematic literature review is carried out to provide understanding on how LLMs have been applied to the classical downstream tasks and to then motivate adaptation of LLMs to idea and hypothesis generation. Furthermore, the study examines techniques applied to customization and knowledge distillation with the aim of contextualizing these methods to solve idea and hypothesis generation. We then explored the limitations of LLM-based research efforts to idea and hypothesis generation. A detailed and technical discussion of the findings of the study is presented, and we provide a high-level novel conceptual framework to describe and summarize our findings. Also, potential insights to combining knowledge graphs, causal inference, logic reasoning and LLMs distillation in idea and hypothesis generation are discussed. Finally, challenges in these research areas on adaptation of LLMs to idea and hypothesis generation are discussed.