Article (Scientific journals)
Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander et al.
2022In Proceedings of the National Academy of Sciences, 119 (44), p. 2203150119
Peer reviewed


Full Text
Publisher postprint (3.5 MB)
Request a copy

All documents in ORBilu are protected by a user license.

Send to


Keywords :
Metascience; Many analysts; Researcher degrees of freedom; Analytical flexibility; Immigration and policy preferences
Abstract :
[en] This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95 % of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
Disciplines :
Sociology & social sciences
Author, co-author :
Breznau, Nate
Rinke, Eike Mark
Wuttke, Alexander
Nguyen, Hung H. V.
Adem, Muna
Adriaans, Jule
Alvarez-Benjumea, Amalia
Andersen, Henrik K.
Auer, Daniel
Azevedo, Flavio
Bahnsen, Oke
Balzer, Dave
Bauer, Gerrit
Bauer, Paul C.
Baumann, Markus
Baute, Sharon
Benoit, Verena
Bernauer, Julian
Berning, Carl
Berthold, Anna
Bethke, Felix S.
Biegert, Thomas
Blinzler, Katharina
Blumenberg, Johannes N.
Bobzien, Licia
Bohman, Andrea
Bol, Thijs
Bostic, Amie
Brzozowska, Zuzanna
Burgdorf, Katharina
Burger, Kaspar
Busch, Kathrin B.
Carlos-Castillo, Juan
Chan, Nathan
Christmann, Pablo
Connelly, Roxanne
Czymara, Christian S.
Damian, Elena
Ecker, Alejandro
Edelmann, Achim
Eger, Maureen A.
Ellerbrock, Simon
Forke, Anna
Forster, Andrea
Gaasendam, Chris
Gavras, Konstantin
Gayle, Vernon
Gessler, Theresa
Gnambs, Timo
Godefroidt, Amélie
Grömping, Max
Groß, Martin
Gruber, Stefan
Gummer, Tobias
Hadjar, Andreas ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Heisig, Jan Paul
Hellmeier, Sebastian
Heyne, Stefanie
Hirsch, Magdalena
Hjerm, Mikael
Hochman, Oshrat
Hövermann, Andreas
Hunger, Sophia
Hunkler, Christian
Huth, Nora
Ignácz, Zsófia S.
Jacobs, Laura
Jacobsen, Jannes
Jaeger, Bastian
Jungkunz, Sebastian
Jungmann, Nils
Kauff, Mathias
Kleinert, Manuel
Klinger, Julia
Kolb, Jan-Philipp
Kołczyńska, Marta
Kuk, John
Kunißen, Katharina
Sinatra, Dafina Kurti
Langenkamp, Alexander
Lersch, Philipp M.
Löbel, Lea-Maria
Lutscher, Philipp
Mader, Matthias
Madia, Joan E.
Malancu, Natalia
Maldonado, Luis
Marahrens, Helge
Martin, Nicole
Martinez, Paul
Mayerl, Jochen
Mayorga, Oscar J.
McManus, Patricia
McWagner, Kyle
Meeusen, Cecil
Meierrieks, Daniel
Mellon, Jonathan
Merhout, Friedolin
Merk, Samuel
Meyer, Daniel
Micheli, Leticia
Mijs, Jonathan
Moya, Cristóbal
Neunhoeffer, Marcel
Nüst, Daniel
Nygård, Olav
Ochsenfeld, Fabian
Otte, Gunnar
Pechenkina, Anna O.
Prosser, Christopher
Raes, Louis
Ralston, Kevin
Ramos, Miguel R.
Roets, Arne
Rogers, Jonathan
Ropers, Guido
Samuel, Robin ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Sand, Gregor
Schachter, Ariela
Schaeffer, Merlin
Schieferdecker, David
Schlueter, Elmar
Schmidt, Regine
Schmidt, Katja M.
Schmidt-Catran, Alexander
Schmiedeberg, Claudia
Schneider, Jürgen
Schoonvelde, Martijn
Schulte-Cloos, Julia
Schumann, Sandy
Schunck, Reinhard
Schupp, Jürgen
Seuring, Julian
Silber, Henning
Sleegers, Willem
Sonntag, Nico
Staudt, Alexander
Steiber, Nadia
Steiner, Nils
Sternberg, Sebastian
Stiers, Dieter
Stojmenovska, Dragana
Storz, Nora
Striessnig, Erich
Stroppe, Anne-Kathrin
Teltemann, Janna
Tibajev, Andrey
Tung, Brian
Vagni, Giacomo
Assche, Jasper Van
Linden, Meta Van Der
Noll, Jolanda Van Der
Hootegem, Arno Van
Vogtenhuber, Stefan
Voicu, Bogdan
Wagemans, Fieke
Wehl, Nadja
Werner, Hannah
Wiernik, Brenton M.
Winter, Fabian
Wolf, Christof
Yamada, Yuki
Zhang, Nan
Ziller, Conrad
Zins, Stefan
Żółtak, Tomasz
More authors (156 more) Less
External co-authors :
Language :
Title :
Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
Publication date :
Journal title :
Proceedings of the National Academy of Sciences
Publisher :
National Academy of Sciences
Volume :
Issue :
Pages :
Peer reviewed :
Peer reviewed
Focus Area :
Migration and Inclusive Societies
Available on ORBilu :
since 31 October 2022


Number of views
409 (277 by Unilu)
Number of downloads
5 (1 by Unilu)

Scopus citations®
Scopus citations®
without self-citations
WoS citations


Similar publications

Contact ORBilu