No full text
Eprint already available on another site (E-prints, Working papers and Research blog)
How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Variability in a Crowdsourced Replication
Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander et al.
2021
 

Files


Full Text
No document available.

Send to



Details



Keywords :
Immigration; Meta-Reliability; Methodology; Noise; Policy Preferences; Political Science; Political Sociology; Replication; Researcher Variability; Secondary Observer Effect; Social and Behavioral Sciences; Sociology
Abstract :
[en] The paper reports findings from a crowdsourced replication. Eighty-four replicator teams attempted to verify results reported in an original study by running the same models with the same data. The replication involved an experimental condition. A “transparent” group received the original study and code, and an “opaque” group received the same underlying study but with only a methods section and description of the regression coefficients without size or significance, and no code. The transparent group mostly verified the original study (95.5%), while the opaque group had less success (89.4%). Qualitative investigation of the replicators’ workflows reveals many causes of non-verification. Two categories of these causes are hypothesized, routine and non-routine. After correcting non-routine errors in the research process to ensure that the results reflect a level of quality that should be present in ‘real-world’ research, the rate of verification was 96.1 in the transparent group and 92.4 in the opaque group. Two conclusions follow: (1) Although high, the verification rate suggests that it would take a minimum of three replicators per study to achieve replication reliability of at least 95 confidence assuming ecological validity in this controlled setting, and (2) like any type of scientific research, replication is prone to errors that derive from routine and undeliberate actions in the research process. The latter suggests that idiosyncratic researcher variability might provide a key to understanding part of the “reliability crisis” in social and behavioral science and is a reminder of the importance of transparent and well documented workflows.
Disciplines :
Sociology & social sciences
Author, co-author :
Breznau, Nate
Rinke, Eike Mark
Wuttke, Alexander
Nguyen, Hung H. V.
Adem, Muna
Adriaans, Jule
Akdeniz, Esra
Alvarez-Benjumea, Amalia
Andersen, Henrik Kenneth
Auer, Daniel
Azevedo, Flavio
Bahnsen, Oke
Bai, Ling
Balzer, Dave
Bauer, Gerrit
Bauer, Paul C.
Baumann, Markus
Baute, Sharon
Benoit, Verena
Bernauer, Julian
Berning, Carl
Berthold, Anna
Bethke, Felix
Biegert, Thomas
Blinzler, Katharina
Blumenberg, Johannes
Bobzien, Licia
Bohman, Andrea
Bol, Thijs
Bostic, Amie
Brzozowska, Zuzanna
Burgdorf, Katharina
Busch, Kathrin
Castillo, Juan Carlos
Chan, Nathan
Christmann, Pablo
Connelly, Roxanne
Czymara, Christian S.
Damian, Elena
Rooij, Eline De
Ecker, Alejandro
Edelmann, Achim
Eder, Christina
Eger, Maureen A.
Ellerbrock, Simon
Forke, Anna
Forster, Andrea
Freire, Danilo
Gaasendam, Chris
Gavras, Konstantin
Gayle, Vernon
Gessler, Theresa
Gnambs, Timo
Godefroidt, Amélie
Grömping, Max
Groß, Martin
Gruber, Stefan
Gummer, Tobias
Hadjar, Andreas  ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Halbherr, Verena
Heisig, Jan Paul
Hellmeier, Sebastian
Heyne, Stefanie
Hirsch, Magdalena
Hjerm, Mikael
Hochman, Oshrat
Höffler, Jan H.
Hövermann, Andreas
Hunger, Sophia
Hunkler, Christian
Huth, Nora
Ignacz, Zsofia
Israel, Sabine
Jacobs, Laura
Jacobsen, Jannes
Jaeger, Bastian
Jungkunz, Sebastian
Jungmann, Nils
Kanjana, Jennifer
Kauff, Mathias
Khan, Salman
Khatua, Sayak
Kleinert, Manuel
Klinger, Julia
Kolb, Jan-Philipp
Kołczyńska, Marta
Kuk, John Seungmin
Kunißen, Katharina
Sinatra, Dafina Kurti
Greinert, Alexander
C. Lee, Robin
Lersch, Philipp M.
Liu, David
Löbel, Lea-Maria
Lutscher, Philipp
Mader, Matthias
Madia, Joan
Malancu, Natalia
Maldonado, Luis
Marahrens, Helge
Martin, Nicole
Martinez, Paul
Mayerl, Jochen
Mayorga, Oscar Jose
McDonnell, Robert Myles
McManus, Patricia
Wagner, Kyle
Meeusen, Cecil
Meierrieks, Daniel
Mellon, Jonathan
Merhout, Friedolin
Merk, Samuel
Meyer, Daniel
Micheli, Leticia
Mijs, Jonathan J. B.
Moya, Cristóbal
Neunhoeffer, Marcel
Nüst, Daniel
Nygård, Olav
Ochsenfeld, Fabian
Otte, Gunnar
Pechenkina, Anna
Pickup, Mark
Prosser, Christopher
Raes, Louis
Ralston, Kevin
Ramos, Miguel
Reichert, Frank
Roets, Arne
Rogers, Jonathan
Ropers, Guido
Samuel, Robin  ;  University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)
Sand, Gregor
Petrarca, Constanza Sanhueza
Schachter, Ariela
Schaeffer, Merlin
Schieferdecker, David
Schlueter, Elmar
Schmidt, Katja
Schmidt, Regine
Schmidt-Catran, Alexander
Schmiedeberg, Claudia
Schneider, Jürgen
Schoonvelde, Martijn
Schulte-Cloos, Julia
Schumann, Sandy
Schunck, Reinhard
Schupp, Jürgen
Seuring, Julian
Silber, Henning
Sleegers, Willem
Sonntag, Nico
Staudt, Alexander
Steiber, Nadia
Steiner, Nils
Sternberg, Sebastian
Sternberg, Sebastian
Stojmenovska, Dragana
Storz, Nora
Striessnig, Erich
Stroppe, Anne-Kathrin
Suchow, Jordan W.
Teltemann, Janna
Tibajev, Andrey
Tung, Brian B.
Vagni, Giacomo
Assche, Jasper Van
Linden, Meta Van Der
Noll, Jolanda Van Der
Hootegem, Arno Van
Vogtenhuber, Stefan
Voicu, Bogdan
Wagemans, Fieke
Wehl, Nadja
Werner, Hannah
Wiernik, Brenton M.
Winter, Fabian
Wolf, Christof
Wu, Cary
Yamada, Yuki
Zakula, Björn
Zhang, Nan
Ziller, Conrad
Zins, Stefan
Żółtak, Tomasz
More authors (176 more) Less
Language :
English
Title :
How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Variability in a Crowdsourced Replication
Publication date :
2021
Publisher :
SocArXiv
Focus Area :
Computational Sciences
Available on ORBilu :
since 09 June 2021

Statistics


Number of views
123 (10 by Unilu)
Number of downloads
0 (0 by Unilu)

OpenCitations
 
1

Bibliography


Similar publications



Contact ORBilu