Improving Reproducibility in AI Research: Four Mechanisms Adopted by JAIR

Background: Lately, the reproducibility of scientific results has become an increasing worry in the scientific community. Several studies show that artificial intelligence research is not spared from reproducibility issues. Objectives: As a pioneer in open and transparent research published on the Internet, the Journal of Artificial Intelligence Research (JAIR) seeks to promote good research practices and close the feedback loop between the original researchers and those reproducing their research. Methods: Four different mechanisms will be adopted immediately by JAIR. These are: 1) reproducibility checklists, 2) structured abstracts, 3) reproducibility badges and 4) reproducibility reports. Results: All authors submitting articles to JAIR fill out a reproducibility checklist and are encouraged to use structured abstracts. Articles that fulfill certain criteria will receive reproducibility badges, and reproducibility reports can be submitted by anyone for any article published in JAIR. Conclusions: We believe that adopting the four mechanisms outlined in this paper will improve the reproducibility of research published in JAIR and thus make a contribution to addressing the broader reproducibility issue in artificial intelligence. We hope that JAIR’s reproducibility initiative will inspire similar efforts at other top-tier journals.