Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

Authors

Institutions

Publication Details

Journal
BMC Biology
Published
2025-02-06
DOI
https://doi.org/10.1186/s12915-024-02101-x
Citations
40
Primary Topic
Species Distribution and Climate Change
Type
article
Field-Weighted Citation Impact
73.53
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Fabiola Iannarilli, Shannon Whelan, Tim S. Doherty, Grégoire Perez et al.
40 citations
BMC Biology
Species Distribution and Climate Change
73.53
article

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Fabiola Iannarilli, Shannon Whelan, Tim S. Doherty, Grégoire Perez, Charles J. W. Carroll, Justin C. Havird, Camille Bordes, Irith Aloni, Daniel L. Weller, Andrew Heaton, Charles A. Martin, Holly M. English, Elliot Gould, Darren J. Parker, Bernd J. Berauer, Biz R. Turnell, James G. Hagan, Tanner J. Howard, Drew Altschul, Raphaël Proulx, Fiona Fidler, Grania Smith, Dominic A. Martin, Penelope Murphy, Malgorzata Lagisz, Kirsty H. Macphie, Elina Takola, Cristian Román‐Palacios, Fabrício Luiz Skupien, Richard J. Telford, Nicolas Schtickzelle, Eduardo Fuentes‐Lillo, Conor C. Taff, Onja H. Razafindratsima, David P. Kochan, Ronja E. M. Wedegärtner, Douglas H. Johnson, Taofeek O. Muraina, Julien Vollering, Lisa Malm, Graham S. Frank, Sepand Riyahi, Paweł Waryszak, Andrea Bonisoli‐Alquati, Garth A. Covernton, Scott W. Yanco, Martin Luquet, Jillian M. Kusch, Kris Sales, Rocío Tarjuelo, J. Stephen Gosnell, Iván Beltrán, Leticia Santos, Tim Schmoll, Alexandra A. de Sousa, Daniela Oliveira de Lima, Sara Gandy, Giacomo Zilio, Andrea Contina, Patrice Pottier, Katharine Keogan, Alison B. Duncan, Claire M. Tortorelli, Ali Güncan, Cédric Zimmer, Ross MacLeod, Pietro B. D’Amelio, Martin Jung, Thomas Botterill‐James, Sabine S. Nooten, Asaf Sadeh, Heather L. Slinn, Esperanza C. Iranzo, Francis van Oordt, Angela Chuang, Michael S. Rosenberg, Mark C. Vanderwel, Emily S. Choy, David C. Ensminger, Skye M. Greenler, Heather A. Hager, Mark McCauley, Natasha Dean Harrison, Felipe Pereira da Rocha, Andrew J. Tanentzap, Timothy Parker, David W. Wolfson, Meghan Barrett, Dominique G. Roche, Diego Simón, Jennifer A. Mortensen, Karen Pedersen, William L. Geary, Shannon Still, Bruno Moreira, Friedrich W. Keppeler, Emily M. Tompkins, Tom Bradfer‐Lawrence, John Fieberg, Tommaso Jucker, Ben Swallow, Luke J. Eberhart‐Phillips, Shane Lindsay, Maaike Griffioen, Thalita Ferreira‐Arruda, Kristian Bell, Jian D. L. Yen, Wolfgang Forstmeier, Svenja Tidau, Don‐Jean Léandri‐Breton, Jonas J. Lembrechts, Rachel L. White, Caitlin P. Mandeville, Daniel J. Larkin, Elizabeth A. Finch, Michael J. Proulx, Casey Youngflesh, Jessica L. Pruett, Erica F. Stuber, Kayla I. Perry, Michael P. Scroggie, Jennifer Bradham, Karen J. Vanderwolf, Trishna Dutta, Alexis Garretson, Agnes B. Olin, Frances J. Griffith, Kate L. Ostevik, Facundo X. Palacio, Matthieu Paquet, Luc F. Bussière, April Robin Martinig, Harrison Goldspiel, Alison Ke, Jessie Lanterman Novotny, Alfredo Sánchez‐Tójar, Marcus Michelangeli, Valerie J. Pasquarella, Sarah Hasnain, Petri T. Niemelä, Yoan Fourcade, Luca Nelli, Ana Payo‐Payo, Xuezhen Ge, Rachel A. Zitomer, Sabrina M. McNew, Alexandra G. Duffy, Eva Lievens, Philippe Marchand, Ralph Dobler, Jake M. Martin, Magdalena M. Mair, Timothée Bonnet, Jared A. Elmore, Stephen M. Ferguson, Jamie Dunning, Thomas Merkling, Diana Carolina Vergara‐Flórez, Peter Korsten, Saheed Olaide Jimoh, Alan Law, Valerio Vitali, Jack A. Brand, Uli Ernst, Peter D. Billman, Suvi Ruuskanen, Jeffrey A. Walker, Kunal Arekar, Martin Brengdahl, Kit Magellan, Shan Kothari, Marco Crotti, Guy F. Sutton, Ana I. García‐Cervigón, Nora Villamil, Ryan J. Weaver, Jake J. Grossman, Elina Mäntylä, Marcus Vinícius Vieira, Oliver Kaltz, Szymon Marian Drobniak, Avery L. Russell, Kaitlin Stack Whitney, Julia Schroeder, Anthony Caravaggi, Martin Bulla, Deusdedith M. Rugemalila, Peter A. Vesk, James Paterson, M. Elise Lauterbur, Jonathan P. Harris, Robert P. Dunn, Caroline L. Shearer, Benjamin Mos, David J. Pascall, Andrew Trlica, Katherine S. Lauck, Michael Manhart, Dustin Gannon, Hugo Thierry, Freya E. Rowland, Jonathan A. Walter, Drew A. Scott, Josh Nightingale, Connor D. Crouch, Kimberly Cressman, Stef Haesen, Eliza M. Grames, Amanda J. Chunco, Shinichi Nakagawa, Nicholas M. Grebe, Cathryn A. Freund, Erin S. McCallum, Francesca Santostefano, Rahel Sollmann, Julie Teresa Shapiro, Patrick Saccone, Kiara L. L'Herpiniere, Bin‐Yan Hsu, Amélie Fontaine, Alexander K. Killion, Joe Atkinson, Ineta Kačergytė, Daniel M. Griffith, Allie E. Schrock, Sarah Goslee, Jonas Kuppler, J. Allan Thomson, Hayden T. Schilling, Gregory M. Zimmerman, Nitika Sharma, Jessica E. M. van der Wal, Paolo Gratton, Diana J. Rennison, David N. Fisher, Pedro Cardoso, Louis Bliard, Heung Ying Janet Chik, Martin Johnsson, Finaritra Randimbiarison, Antje Girndt, Marcus Schmidt, Stefano Mammola, Murray P. Cox, Christopher J. Thawley, Therese A. Catanach, Sara E. Campbell, Scott J. Meiners, María Moirón, Dylan Gomes, Christopher M. Baker, Timm F. Döbert, Esteban Fernández‐Juricic, Simon C. Griffith, Veronarindra Ramananjato, Nick L. Schultz, Clint D. Kelly, Michael I. Sitvarin, Carles Alcaráz, Evan A. Fiorenza, Michael J. Roast, Lara Urban, Xanthe J. Walker, Erik N. K. Iverson, Marc Gilles, Aneesh P. H. Bose, Daniel Gliksman, Jens Van Eeckhoven, Jesse Jorna, Megan Good, Adam C. Schneider, Ettore Camerlenghi, Juliana Vélez, Brian C. Verrelli, Charlie Blake, Sergio Nolazco, Benjamin Michael Marshall, Chris L. Organ, Gustav Nilsonne, Jeremy A. Smith, Suleiman Kehinde Bello, Kristen Lalla, Dongmin Kim, Jeff W. Atkins, Michael G. Bertram, Daniel G. Hamilton, Federico Riva, Courtney L. Larson, Charly Géron, Melissa Anna Boyd, Mahmoud Medhat Elsherif, Jessica K. Abbott, Stephen M. Salazar, A. Dobson, Hannah Fraser, Sarah A. Boyle, Leonardo Leite Ferraz de Campos, Pablo Salmón, Bethany L. Clark, Luis Alejandro Aguirre, K. Michelle Vanderwel, Andrew Wood, Stijn Van de Vondel, Xuan Chen, S. F. Blake, Jessica Ross, Laura Milena Manrique‐Garzón, María Laura Herrera-Chaustre
article en
40 citations

Abstract

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

BMC BiologyVol. 23(1)
Brigham Young University (US), Université Claude Bernard Lyon 1 (FR), Wesleyan University (US), University College Dublin (IE), Linköping University (SE), Rutgers, The State University of New Jersey (US), University of California, Riverside (US), Universidad de Valladolid (ES), University of Hohenheim (DE), University of Rome Tor Vergata (IT), Rhodes College (US), University of Bern (CH), Missouri State University (US), Boise State University (US), University of Stirling (GB), Universidad Complutense de Madrid (ES), Tufts University (US), Utah State University (US), Universidade Federal do Rio de Janeiro (BR), Binghamton University (US), Ben-Gurion University of the Negev (IL), Jagiellonian University (PL), The Nature Conservancy (US), University of Wyoming (US), University of South Wales (GB), Centre National de la Recherche Scientifique (FR), The University of Sydney (AU), University of Vienna (AT), University of Tasmania (AU), University of Minnesota (US), CAB International (GB), Memorial University of Newfoundland (CA), Wofford College (US), Centre de Coopération Internationale en Recherche Agronomique pour le Développement (FR), Agricultural Research Service (US), Oregon State University (US), International Institute for Applied Systems Analysis (AT), Arthur Rylah Institute for Environmental Research (AU), Minnesota Department of Natural Resources (US), NHS Lothian (GB), Office for National Statistics (GB), California Department of Fish and Wildlife (US), United States Department of Agriculture (US), University of Helsinki (FI), St. Norbert College (US), Consejo Superior de Investigaciones Científicas (ES), Royal Society for the Protection of Birds (GB), University of Indianapolis (US), University of Wisconsin–Madison (US), Harvard University (US), Helmholtz Centre for Environmental Research (DE), University of Antananarivo (MG), North Carolina State University (US), University of Arizona (US), Austrian Academy of Sciences (AT), Whitman College (US), Bar-Ilan University (IL), University of Connecticut (US), University of Lisbon (PT), Baruch College (US), Université de Pau et des Pays de l'Adour (FR), Lake Superior State University (US), University of Antwerp (BE), Federation University (AU), Deakin University (AU), University of Veterinary Medicine Vienna (AT), University of Waterloo (CA), Eastern Illinois University (US), University of Auckland (NZ), University of Alberta (CA), Inserm (FR), Rochester Institute of Technology (US), University of Turku (FI), University of South Carolina (US), University of Cape Town (ZA), Université du Québec à Montréal (CA), State Street (United States) (US), École Pratique des Hautes Études (FR), Aberystwyth University (GB), University of California, Los Angeles (US), Bangor University (GB), Universidad de Los Andes (CO), Austral University of Chile (CL), The University of Queensland (AU), University of Iceland (IS), The University of Melbourne (AU), University of St Andrews (GB), University of Groningen (NL), Elon University (US), Duke University (US), Iowa State University (US), University of Rhode Island (US), The University of Western Australia (AU), Western Norway University of Applied Sciences (NO), Universidad de la República de Uruguay (UY), Universidad Rey Juan Carlos (ES), Virginia Commonwealth University (US), King Abdulaziz University (SA), University of Toronto (CA), University of Southern Queensland (AU), Institute of Technology of Cambodia (KH), Lund University (SE), University of Konstanz (DE), Université du Québec en Abitibi-Témiscamingue (CA), University of Hull (GB), University of Regina (CA), University of Aberdeen (GB), Universität Ulm (DE), Florida International University (US), Université Paris-Est Créteil (FR), Université de Montpellier (FR), Bielefeld University (DE), Middle East Technical University (TR), University of Zurich (CH), Northern Arizona University (US), Rhodes University (ZA), University of California, Irvine (US), Aarhus University (DK), Norwegian University of Science and Technology (NO), Cornell University (US), Czech University of Life Sciences Prague (CZ), Purdue University West Lafayette (US), Montana State University (US), University of Exeter (GB), University of Cambridge (GB), University of Massachusetts Amherst (US), St. Olaf College (US), University of Würzburg (DE), University of Michigan (US), Wells National Estuarine Research Reserve (US), Whitney Museum of American Art (US), New South Wales Department of Primary Industries (AU), The University of Texas Rio Grande Valley (US), Karolinska Institutet (SE), Swedish University of Agricultural Sciences (SE), Helmholtz Zentrum München (DE), Université Rennes 2 (FR), Universidade Federal da Fronteira Sul (BR), Technische Universität Darmstadt (DE), UNSW Sydney (AU), University of Saskatchewan (CA), Yale University (US), University of Florida (US), Vestlandsforsking (NO), University of Bristol (GB), University of California San Diego (US), Agricultural Research Organization (IL), Sorbonne Université (FR), University of Oxford (GB), Universidade Federal de Santa Catarina (BR), Max Planck Institute for Ornithology (DE), Institute for Interdisciplinary Mountain Research (AT), Université Sorbonne Paris Nord (FR), Mammoth Hospital (US), Animal, Santé, Territoires, Risques et Ecosystèmes (FR), Centre d'Etudes Biologiques de Chizé (FR), Institute of Ecology and Biodiversity (CL), Institut des Sciences de l'Evolution de Montpellier (FR), Centre d'Investigacions sobre Desertificació (ES), Nutrition Metabolism Aquaculture (FR), Leibniz Centre for Agricultural Landscape Research (DE), Southern Research Station (US), Public Health Scotland (GB), Water Research Institute (IT), Documentation Center of Cambodia (KH), County Administrative Board (SE), Laboratoire d’Ethologie Expérimentale et Comparée (FR), Dana-Farber/Harvard Cancer Center (US), Institut d'écologie et des sciences de l'environnement de Paris (FR), MRC Biostatistics Unit (GB), Agence Nationale de Sécurité Sanitaire de l’Alimentation, de l’Environnement et du Travail (FR), Institute of Avian Research (DE), BirdLife international (GB), Institut Jean Lamour (FR), European Forest Institute (DE), National Research Council (IT), Environmental Earth Sciences (AU), The FRAM Centre (NO), Ecosystem Sciences (AU), Institut de Recherche pour le Développement (FR), University of Bergen (NO), University of Southern Mississippi (US), University College London (GB), University of Wisconsin–La Crosse (US), Wake Forest University (US), Imperial College London (GB), McGill University (CA), San Jose State University (US), University of Bath (GB), The Ohio State University (US), University of Bayreuth (DE), Indiana University – Purdue University Indianapolis (US), Université de Rennes (FR), Monash University (AU), University of Neuchâtel (CH), Indian Institute of Science Bangalore (IN), Liverpool John Moores University (GB), Université du Québec à Trois-Rivières (CA), Ordu University (TR), Trent University (CA), Universidade Federal de Goiás (BR), Université de Montréal (CA), University of Brighton (GB), Drexel University (US), University of Southern Maine (US), University of Göttingen (DE), University of Tennessee at Knoxville (US), Wilfrid Laurier University (CA), Hiram College (US), Technische Universität Dresden (DE), University of Arkansas at Fayetteville (US), University of Glasgow (GB), University of Maine (US), University of Birmingham (GB), University of Guelph (CA), Clemson University (US), University of California, Davis (US), Virginia Tech (US), British Trust for Ornithology (GB), The University of Texas at Austin (US), Vrije Universiteit Amsterdam (NL), Michigan State University (US), Universidad Nacional de La Plata (AR), Southern Illinois University Edwardsville (US), University of Gothenburg (SE), University of Hong Kong (HK), University of Plymouth (GB), Université de Lorraine (FR), Umeå University (SE), Bath Spa University (GB), University of Colorado Denver (US), Colorado State University (US), Salisbury University (US), University of Jyväskylä (FI), University of California, Berkeley (US), UCLouvain (BE), McMaster University (CA), University of Edinburgh (GB), California State Polytechnic University (US), Macquarie University (AU), KU Leuven (BE)
Openalex Percentile: Top 0%
Species Distribution and Climate Change
73.53
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.