Reproducibility of in vivo electrophysiological measurements in mice

Understanding brain function relies on the collective work of many labs generating reproducible results. However, reproducibility has not been systematically assessed within the context of electrophysiological recordings during cognitive behaviors. To address this, we formed a multi-lab collaboration using a shared, open-source behavioral task and experimental apparatus. Experimenters in 10 laboratories repeatedly targeted Neuropixels probes to the same location (spanning secondary visual areas, hippocampus, and thalamus) in mice making decisions; this generated a total of 121 experimental replicates, a unique dataset for evaluating reproducibility of electrophysiology experiments. Despite standardizing both behavioral and electrophysiological procedures, some experimental outcomes were highly variable. A closer analysis uncovered that variability in electrode targeting hindered reproducibility, as did the limited statistical power of some routinely used electrophysiological analyses, such as single-neuron tests of modulation by individual task parameters. Reproducibility was enhanced by histological and electrophysiological quality-control criteria. Our observations suggest that data from systems neuroscience is vulnerable to a lack of reproducibility, but that across-lab standardization, including metrics we propose, can serve to mitigate this.

Authors

Institutions

Publication Details

Journal
eLife
Published
2024-10-31
DOI
https://doi.org/10.7554/elife.100840
Citations
14
Primary Topic
Neuroscience and Neuropharmacology Research
Type
article
Field-Weighted Citation Impact
2.15
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Reproducibility of in vivo electrophysiological measurements in mice

Nicholas A. Steinmetz, Kelly Buchanan, Zeinab Mohammadi, Anne E Urai et al.
14 citations
eLife
Neuroscience and Neuropharmacology Research
2.15
article

Reproducibility of in vivo electrophysiological measurements in mice

Nicholas A. Steinmetz, Kelly Buchanan, Zeinab Mohammadi, Anne E Urai, Miles J. Wells, Steven J. West, Robert A. A. Campbell, Jean‐Paul Noel, Kush Banga, Cyrille Rossant, Christopher Krasniak, Matteo Carandini, Yizi Zhang, Guido T. Meijer, Jai Bhagat, Noam Roth, Matthew R Whiteway, Nathaniel J Miska, M. Felicia Davatolhagh, Julius Benson, Julia M. Huntenburg, Berk Gerçek, Gaëlle Chapuis, Anup Khanal, Christopher Langfield, Anne K. Churchland, Mayo Faulkner, Liam Paninski, Sebastian A. Bruijns, Ilana B. Witten, Alejandro Pan-Vazquez, Olivier Winter, Marsa Taheri, Michael Schartner, Dan Biderman, Daniel Birman, Fei Hu, Cole Hurwitz, Niccolò Bonacchi, Karolina Socha, Petrina Lau, Karel Svoboda, Shuqi Wang, Neil E. Mackenzie, Hyun Dong Lee
article en
14 citations

Abstract

Understanding brain function relies on the collective work of many labs generating reproducible results. However, reproducibility has not been systematically assessed within the context of electrophysiological recordings during cognitive behaviors. To address this, we formed a multi-lab collaboration using a shared, open-source behavioral task and experimental apparatus. Experimenters in 10 laboratories repeatedly targeted Neuropixels probes to the same location (spanning secondary visual areas, hippocampus, and thalamus) in mice making decisions; this generated a total of 121 experimental replicates, a unique dataset for evaluating reproducibility of electrophysiology experiments. Despite standardizing both behavioral and electrophysiological procedures, some experimental outcomes were highly variable. A closer analysis uncovered that variability in electrode targeting hindered reproducibility, as did the limited statistical power of some routinely used electrophysiological analyses, such as single-neuron tests of modulation by individual task parameters. Reproducibility was enhanced by histological and electrophysiological quality-control criteria. Our observations suggest that data from systems neuroscience is vulnerable to a lack of reproducibility, but that across-lab standardization, including metrics we propose, can serve to mitigate this.

eLifeVol. 13
Champalimaud Foundation (PT), University of Geneva (CH), Leiden University (NL), University of California, Los Angeles (US), Laboratoire d'Informatique Fondamentale de Lille (FR), Princeton University (US), University of Washington (US), Cold Spring Harbor Laboratory (US), Sainsbury Laboratory (GB), University of California System (US), Allen Institute (US), Allen Institute for Neural Dynamics (US), University College London (GB), New York University (US), Columbia University (US), University of California, Berkeley (US)
Openalex Percentile: Top 12%
Neuroscience and Neuropharmacology Research
2.15
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.