Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

[en] This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95 % of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Disciplines :

Sociology & social sciences

Author, co-author :

Breznau, Nate

Rinke, Eike Mark

Wuttke, Alexander

Nguyen, Hung H. V.

Adem, Muna

Adriaans, Jule

Alvarez-Benjumea, Amalia

Andersen, Henrik K.

Auer, Daniel

Azevedo, Flavio

More authors (156 more)

External co-authors :

yes

Language :

English

Title :

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Publication date :

2022

Journal title :

Proceedings of the National Academy of Sciences

ISSN :

0027-8424

eISSN :

1091-6490

Publisher :

National Academy of Sciences

Volume :

119

Issue :

Pages :

e2203150119

Peer reviewed :

Peer reviewed

Focus Area :

Migration and Inclusive Societies

Additional URL :

https://www.pnas.org/doi/full/10.1073/pnas.2203150119

Available on ORBilu :

since 31 October 2022

Statistics

Number of views

409 (277 by Unilu)

Number of downloads

5 (1 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

WoS citations^™

Bibliography

M. Solomon, Social Empiricism (MIT Press, 2007).
N. Oreskes, Why Trust Science? (Princeton University Press, 2019).
C. F. Camerer et al., Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2, 637–644 (2018).
Open Science Collaboration, PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
S. Ritchie, Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (Metropolitan Books, 2020).
A. B. Sørensen, The structural basis of social inequality. Am. J. Sociol. 101, 1333–1365 (1996).
B. S. Frey, Publishing as prostitution? Choosing between one’s own ideas and academic success. Public Choice 116, 205–223 (2003).
A. Gelman, E. Loken, The statistical crisis in science. Am. Sci. 102, 460 (2014).
A. Orben, A. K. Przybylski, The association between adolescent well-being and digital technology use. Nat. Hum. Behav. 3, 173–182 (2019).
M. Del Giudice, S. Gangestad, A traveler’s guide to the multiverse: Promises, pitfalls, and a framework for the evaluation of analytic decisions. Adv. Methods Pract. Psychol. Sci., 10.1177/ 2515245920954925 (2021).
J. P. Simmons, L. D. Nelson, U. Simonsohn, False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).
U. Schimmack, A meta-psychological perspective on the decade of replication failures in social psychology. Can. Psychol. 61, 364–376 (2020).
J. Freese, D. Peterson, The emergence of statistical objectivity: Changing ideas of epistemic vice and virtue in science. Sociol. Theory 36, 289–313 (2018).
R. Silberzahn et al., Many analysts, one data set: Making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. 1, 337–356 (2018).
G. Dutilh et al., The quality of response time data inference: A blinded, collaborative assessment of the validity of cognitive models. Psychon. Bull. Rev. 26, 1051–1069 (2019).
J. A. Bastiaansen et al., Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology. J. Psychosom. Res. 137, 110211 (2020).
R. Botvinik-Nezer et al., Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).
A. J. Menkveld et al., Non-standard errors. Social Science Research Network [Preprint] (2022). https://doi.org/10.2139/ssrn.3961574 (Accessed 4 January 2022).
D. Brady, R. Finnigan, Does immigration undermine public support for social policy? Am. Sociol. Rev. 79, 17–42 (2014).
K. Auspurg, J. Br€uderl, Has the credibility of the social sciences been credibly destroyed? Reanalyzing the “Many Analysts, One Data Set” Project. Socius 86, 532–565 (2021).
I. Lundberg, R. Johnson, B. M. Stewart, What is your estimand? Defining the target quantity connects statistical evidence to theory. Am. Sociol. Rev. 86, 532–565 (2021).
A. Alesina, E. Glaeser, Why are welfare states in the US and Europe so different? What do we learn? Horizons Strategiques 2, 51–61 (2006).
A. Alesina, E. Glaeser, Fighting Poverty in the US and Europe: A World of Difference (Oxford University Press, 2004).
A. Alesina, E. Murard, H. Rapoport, Immigration and preferences for redistribution in Europe. J. Econ. Geogr. 21, 925–954 (2021).
M. A. Eger, Even in Sweden: The effect of immigration on support for welfare state spending. Eur. Sociol. Rev. 26, 203–217 (2010).
J. C. Garand, P. Xu, B. C. Davis, Immigration attitudes and support for the welfare state in the American mass public. Am. J. Pol. Sci. 61, 146–162 (2017).
B. Burgoon, Immigration, integration, and support for redistribution in Europe. World Polit. 66, 365–405 (2014).
J. Alt, T. Iversen, Inequality, labor market segmentation, and preferences for redistribution. Am. J. Pol. Sci. 61, 21–36 (2017).
S. Pardos-Prado, C. Xena, Immigration and support for social policy: An experimental comparison of universal and means-tested programs. Political Sci. Res. Methods 7, 717–735 (2019).
M. L. Bryan, S. P. Jenkins, Multilevel modelling of country effects: A cautionary tale. Eur. Sociol. Rev. 32, 3–22 (2016).
M. Schweinsberg et al., Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organ. Behav. Hum. Decis. Process. 165, 228–249 (2021).
M. A. Eger, N. Breznau, Immigration and the welfare state: A cross-regional analysis of European welfare attitudes. Int. J. Comp. Sociol. 58, 440–463 (2017).
N. Breznau et al., The Hidden Universe of Data-Analysis. GitHub. https://github.com/nbreznau/CRI. Deposited 12 April 2021.
N. Breznau et al., The Crowdsourced Replication Initiative Participant Survey. Harvard Dataverse. https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/UUP8CX. Deposited 24 March 2021.