Chronological Evaluation of Emerging Methodology Extraction from AI Literature

The rapid growth of scientific literature, particularly in Artificial Intelligence (AI), has resulted in the frequent introduction of new methodologies and evolving meanings for existing terms. However, in AI, the fast-paced development poses challenges for researchers in keeping up with recent methodologies (e.g., ‘R-CNN’, ‘ELMo’), crucial for tasks such as idea generation, baseline selection, and peer review. Existing automated methods for extracting key concepts from research articles are limited by their static evaluation setups and lack of adaptability to newer terminology. Addressing these limitations, this paper proposes a factored approach leveraging category information (e.g., ‘NLP’, ‘RL’) to methodology extraction, which utilizes domain-specific partitioning either by input or label space. We conduct experiments under a zero- and few-shot setup and introduce a chronological evaluation framework. Our frameworks learn incrementally using silver-standard data from previous predictions to adapt to recent data. Empirical results demonstrate that our proposed method outperforms prior approaches like SciREX in effectively identifying emerging methodology names in scientific literature. Experiments on a simulated setup using historical and incrementally updated data show that our approach improves future predictions on newer papers, outperforming baselines by up to 9.257% in the few-shot setting.