Generating and choosing research questions with AI agents

Judgements of candidate research questions are noisy, and practitioner checklists such as FINER have not been tested against research outcomes. We tested methods for generating and choosing research questions in one field, the observable behaviour of small open language models, where every question could be piloted in minutes. Three Claude models generated questions by four methods (168 after merging near-duplicates). Four selection methods scored every question before any outcome was known; every question then had a full pilot on three small models and a fixed literature search. The primary outcome was a new effect detected in a pilot: the full pilot met a pass test fixed in advance, and the search found no earlier report of the effect in the same direction. Of 168 questions, 37 (22%) gave one; an audit after the results found earlier reports for 8 of these 37, without changing the comparison of selectors. Critique and forecast (two models named each question's most likely flaw, then three models forecast its outcome) predicted this outcome better than chance (area under the ROC curve, AUROC, 0.684; 95% interval 0.59 to 0.78); in an exploratory analysis, almost all of this came from the forecast that the pilot would pass. A screening pilot, a short pilot built independently of the full one, did about as well (0.712) and predicted passing better (0.807 against 0.622). A FINER checklist scored by one model (0.514) and random scores (0.550) did not beat chance. Plain prompting and transfer from distant fields gave the most new effects (28% and 27% of their questions) and gap mapping from recent abstracts the fewest (8%), mostly because its earlier reports were abstracts its generators had read; rates also differed by generating model (19% to 33%), though these differences could be chance. The method built from these results generates with several models, screens every question with a short pilot, and checks novelty with a recorded search. The components were tested separately in one field, Claude models did all the generating and judging, and a new effect need not be important. Where pilots are cheap, the results favour running short pilots of candidate questions over rating them, and searching the literature over asking models whether a question is new.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24208708
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Generating and choosing research questions with AI agents

Brendan Barlow
3 links
Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
preprint

Generating and choosing research questions with AI agents

Brendan Barlow
preprint en

Abstract

Judgements of candidate research questions are noisy, and practitioner checklists such as FINER have not been tested against research outcomes. We tested methods for generating and choosing research questions in one field, the observable behaviour of small open language models, where every question could be piloted in minutes. Three Claude models generated questions by four methods (168 after merging near-duplicates). Four selection methods scored every question before any outcome was known; every question then had a full pilot on three small models and a fixed literature search. The primary outcome was a new effect detected in a pilot: the full pilot met a pass test fixed in advance, and the search found no earlier report of the effect in the same direction. Of 168 questions, 37 (22%) gave one; an audit after the results found earlier reports for 8 of these 37, without changing the comparison of selectors. Critique and forecast (two models named each question's most likely flaw, then three models forecast its outcome) predicted this outcome better than chance (area under the ROC curve, AUROC, 0.684; 95% interval 0.59 to 0.78); in an exploratory analysis, almost all of this came from the forecast that the pilot would pass. A screening pilot, a short pilot built independently of the full one, did about as well (0.712) and predicted passing better (0.807 against 0.622). A FINER checklist scored by one model (0.514) and random scores (0.550) did not beat chance. Plain prompting and transfer from distant fields gave the most new effects (28% and 27% of their questions) and gap mapping from recent abstracts the fewest (8%), mostly because its earlier reports were abstracts its generators had read; rates also differed by generating model (19% to 33%), though these differences could be chance. The method built from these results generates with several models, screens every question with a short pilot, and checks novelty with a recorded search. The components were tested separately in one field, Claude models did all the generating and judging, and a new effect need not be important. Where pilots are cheap, the results favour running short pilots of candidate questions over rating them, and searching the literature over asking models whether a question is new.

Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon