No full text
Eprint already available on another site (E-prints, Working papers and Research blog)
Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander et al.
2021
 

Files


Full Text
No document available.

Send to



Details



Keywords :
Analytical Flexibility; Crowdsourced Replication Initiative; Crowdsourcing; Economics; Garden of Forking Paths; Immigration; Many Analysts; Meta-Science; Noise; Other Social and Behavioral Sciences; Political Science; Psychology; Researcher Degrees of Freedom; Researcher Variability; Social and Behavioral Sciences; Social Policy; Sociology
Abstract :
[en] How does noise generated by researcher decisions undermine the credibility of science? We test this by observing all decisions made among 73 research teams as they independently conduct studies on the same hypothesis with identical starting data. We find excessive variation of outcomes. When combined, the 107 observed research decisions taken across teams explained at most 2.6 of the total variance in effect sizes and 10 of the deviance in subjective conclusions. Expertise, prior beliefs and attitudes of the researchers explain even less. Each model deployed to test the hypothesis was unique, which highlights a vast universe of research design variability that is normally hidden from view and suggests humility when presenting and interpreting scientific findings.
Disciplines :
Sociology & social sciences
Author, co-author :
Breznau, Nate
Rinke, Eike Mark
Wuttke, Alexander
Adem, Muna
Adriaans, Jule
Alvarez-Benjumea, Amalia
Andersen, Henrik Kenneth
Auer, Daniel
Azevedo, Flavio
Bahnsen, Oke
Balzer, Dave
Bauer, Gerrit
Bauer, Paul C.
Baumann, Markus
Baute, Sharon
Benoit, Verena
Bernauer, Julian
Berning, Carl
Berthold, Anna
Bethke, Felix
Biegert, Thomas
Blinzler, Katharina
Blumenberg, Johannes
Bobzien, Licia
Bohman, Andrea
Bol, Thijs
Bostic, Amie
Brzozowska, Zuzanna
Burgdorf, Katharina
Busch, Kathrin
Castillo, Juan Carlos
Chan, Nathan
Christmann, Pablo
Connelly, Roxanne
Czymara, Christian S.
Damian, Elena
Ecker, Alejandro
Edelmann, Achim
Eger, Maureen A.
Ellerbrock, Simon
Forke, Anna
Forster, Andrea
Gaasendam, Chris
Gavras, Konstantin
Gayle, Vernon
Gessler, Theresa
Gnambs, Timo
Godefroidt, Amélie
Grömping, Max
Groß, Martin
Gruber, Stefan
Gummer, Tobias
Hadjar, Andreas ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Heisig, Jan Paul
Hellmeier, Sebastian
Heyne, Stefanie
Hirsch, Magdalena
Hjerm, Mikael
Hochman, Oshrat
Hövermann, Andreas
Hunger, Sophia
Hunkler, Christian
Huth, Nora
Ignacz, Zsofia
Jacobs, Laura
Jacobsen, Jannes
Jaeger, Bastian
Jungkunz, Sebastian
Jungmann, Nils
Kauff, Mathias
Kleinert, Manuel
Klinger, Julia
Kolb, Jan-Philipp
Kołczyńska, Marta
Kuk, John Seungmin
Kunißen, Katharina
Sinatra, Dafina Kurti
Greinert, Alexander
Lersch, Philipp M.
Löbel, Lea-Maria
Lutscher, Philipp
Mader, Matthias
Madia, Joan
Malancu, Natalia
Maldonado, Luis
Marahrens, Helge
Martin, Nicole
Martinez, Paul
Mayerl, Jochen
Mayorga, Oscar Jose
McManus, Patricia
Wagner, Kyle
Meeusen, Cecil
Meierrieks, Daniel
Mellon, Jonathan
Merhout, Friedolin
Merk, Samuel
Meyer, Daniel
Micheli, Leticia
Mijs, Jonathan J. B.
Moya, Cristóbal
Neunhoeffer, Marcel
Nüst, Daniel
Nygård, Olav
Ochsenfeld, Fabian
Otte, Gunnar
Pechenkina, Anna
Prosser, Christopher
Raes, Louis
Ralston, Kevin
Ramos, Miguel
Roets, Arne
Rogers, Jonathan
Ropers, Guido
Samuel, Robin  ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Sand, Gregor
Schachter, Ariela
Schaeffer, Merlin
Schieferdecker, David
Schlueter, Elmar
Schmidt, Katja M.
Schmidt, Regine
Schmidt-Catran, Alexander
Schmiedeberg, Claudia
Schneider, Jürgen
Schoonvelde, Martijn
Schulte-Cloos, Julia
Schumann, Sandy
Schunck, Reinhard
Schupp, Jürgen
Seuring, Julian
Silber, Henning
Sleegers, Willem
Sonntag, Nico
Staudt, Alexander
Steiber, Nadia
Steiner, Nils
Sternberg, Sebastian
Stiers, Dieter
Stojmenovska, Dragana
Storz, Nora
Striessnig, Erich
Stroppe, Anne-Kathrin
Teltemann, Janna
Tibajev, Andrey
Tung, Brian B.
Vagni, Giacomo
Assche, Jasper Van
Linden, Meta Van Der
Noll, Jolanda Van Der
Hootegem, Arno Van
Vogtenhuber, Stefan
Voicu, Bogdan
Wagemans, Fieke
Wehl, Nadja
Werner, Hannah
Wiernik, Brenton M.
Winter, Fabian
Wolf, Christof
Yamada, Yuki
Zhang, Nan
Ziller, Conrad
Zins, Stefan
Żółtak, Tomasz
Nguyen, Hung H. V.
More authors (156 more) Less
Language :
English
Title :
Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
Publication date :
2021
Publisher :
MetaArXiv
Focus Area :
Computational Sciences
Available on ORBilu :
since 09 June 2021

Statistics


Number of views
195 (25 by Unilu)
Number of downloads
0 (0 by Unilu)

OpenCitations
 
9

Bibliography


Similar publications



Contact ORBilu