Reference : How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Va...
E-prints/Working papers : Already available on another site
Social & behavioral sciences, psychology : Sociology & social sciences
Computational Sciences
http://hdl.handle.net/10993/47447
How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Variability in a Crowdsourced Replication
English
Breznau, Nate [> >]
Rinke, Eike Mark [> >]
Wuttke, Alexander [> >]
Nguyen, Hung H. V. [> >]
Adem, Muna [> >]
Adriaans, Jule [> >]
Akdeniz, Esra [> >]
Alvarez-Benjumea, Amalia [> >]
Andersen, Henrik Kenneth [> >]
Auer, Daniel [> >]
Azevedo, Flavio [> >]
Bahnsen, Oke [> >]
Bai, Ling [> >]
Balzer, Dave [> >]
Bauer, Gerrit [> >]
Bauer, Paul C. [> >]
Baumann, Markus [> >]
Baute, Sharon [> >]
Benoit, Verena [> >]
Bernauer, Julian [> >]
Berning, Carl [> >]
Berthold, Anna [> >]
Bethke, Felix [> >]
Biegert, Thomas [> >]
Blinzler, Katharina [> >]
Blumenberg, Johannes [> >]
Bobzien, Licia [> >]
Bohman, Andrea [> >]
Bol, Thijs [> >]
Bostic, Amie [> >]
Brzozowska, Zuzanna [> >]
Burgdorf, Katharina [> >]
Burger, Kaspar [> >]
Busch, Kathrin [> >]
Castillo, Juan Carlos [> >]
Chan, Nathan [> >]
Christmann, Pablo [> >]
Connelly, Roxanne [> >]
Czymara, Christian S. [> >]
Damian, Elena [> >]
Rooij, Eline De [> >]
Ecker, Alejandro [> >]
Edelmann, Achim [> >]
Eder, Christina [> >]
Eger, Maureen A. [> >]
Ellerbrock, Simon [> >]
Forke, Anna [> >]
Forster, Andrea [> >]
Freire, Danilo [> >]
Gaasendam, Chris [> >]
Gavras, Konstantin [> >]
Gayle, Vernon [> >]
Gessler, Theresa [> >]
Gnambs, Timo [> >]
Godefroidt, Amélie [> >]
Grömping, Max [> >]
Groß, Martin [> >]
Gruber, Stefan [> >]
Gummer, Tobias [> >]
Hadjar, Andreas mailto [University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)]
Halbherr, Verena [> >]
Heisig, Jan Paul [> >]
Hellmeier, Sebastian [> >]
Heyne, Stefanie [> >]
Hirsch, Magdalena [> >]
Hjerm, Mikael [> >]
Hochman, Oshrat [> >]
Höffler, Jan H. [> >]
Hövermann, Andreas [> >]
Hunger, Sophia [> >]
Hunkler, Christian [> >]
Huth, Nora [> >]
Ignacz, Zsofia [> >]
Israel, Sabine [> >]
Jacobs, Laura [> >]
Jacobsen, Jannes [> >]
Jaeger, Bastian [> >]
Jungkunz, Sebastian [> >]
Jungmann, Nils [> >]
Kanjana, Jennifer [> >]
Kauff, Mathias [> >]
Khan, Salman [> >]
Khatua, Sayak [> >]
Kleinert, Manuel [> >]
Klinger, Julia [> >]
Kolb, Jan-Philipp [> >]
Kołczyńska, Marta [> >]
Kuk, John Seungmin [> >]
Kunißen, Katharina [> >]
Sinatra, Dafina Kurti [> >]
Greinert, Alexander [> >]
C. Lee, Robin [> >]
Lersch, Philipp M. [> >]
Liu, David [> >]
Löbel, Lea-Maria [> >]
Lutscher, Philipp [> >]
Mader, Matthias [> >]
Madia, Joan [> >]
Malancu, Natalia [> >]
Maldonado, Luis [> >]
Marahrens, Helge [> >]
Martin, Nicole [> >]
Martinez, Paul [> >]
Mayerl, Jochen [> >]
Mayorga, Oscar Jose [> >]
McDonnell, Robert Myles [> >]
McManus, Patricia [> >]
Wagner, Kyle [> >]
Meeusen, Cecil [> >]
Meierrieks, Daniel [> >]
Mellon, Jonathan [> >]
Merhout, Friedolin [> >]
Merk, Samuel [> >]
Meyer, Daniel [> >]
Micheli, Leticia [> >]
Mijs, Jonathan J. B. [> >]
Moya, Cristóbal [> >]
Neunhoeffer, Marcel [> >]
Nüst, Daniel [> >]
Nygård, Olav [> >]
Ochsenfeld, Fabian [> >]
Otte, Gunnar [> >]
Pechenkina, Anna [> >]
Pickup, Mark [> >]
Prosser, Christopher [> >]
Raes, Louis [> >]
Ralston, Kevin [> >]
Ramos, Miguel [> >]
Reichert, Frank [> >]
Roets, Arne [> >]
Rogers, Jonathan [> >]
Ropers, Guido [> >]
Samuel, Robin mailto [University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Social Sciences (DSOC)]
Sand, Gregor [> >]
Petrarca, Constanza Sanhueza [> >]
Schachter, Ariela [> >]
Schaeffer, Merlin [> >]
Schieferdecker, David [> >]
Schlueter, Elmar [> >]
Schmidt, Katja [> >]
Schmidt, Regine [> >]
Schmidt-Catran, Alexander [> >]
Schmiedeberg, Claudia [> >]
Schneider, Jürgen [> >]
Schoonvelde, Martijn [> >]
Schulte-Cloos, Julia [> >]
Schumann, Sandy [> >]
Schunck, Reinhard [> >]
Schupp, Jürgen [> >]
Seuring, Julian [> >]
Silber, Henning [> >]
Sleegers, Willem [> >]
Sonntag, Nico [> >]
Staudt, Alexander [> >]
Steiber, Nadia [> >]
Steiner, Nils [> >]
Sternberg, Sebastian [> >]
Sternberg, Sebastian [> >]
Stojmenovska, Dragana [> >]
Storz, Nora [> >]
Striessnig, Erich [> >]
Stroppe, Anne-Kathrin [> >]
Suchow, Jordan W. [> >]
Teltemann, Janna [> >]
Tibajev, Andrey [> >]
Tung, Brian B. [> >]
Vagni, Giacomo [> >]
Assche, Jasper Van [> >]
Linden, Meta Van Der [> >]
Noll, Jolanda Van Der [> >]
Hootegem, Arno Van [> >]
Vogtenhuber, Stefan [> >]
Voicu, Bogdan [> >]
Wagemans, Fieke [> >]
Wehl, Nadja [> >]
Werner, Hannah [> >]
Wiernik, Brenton M. [> >]
Winter, Fabian [> >]
Wolf, Christof [> >]
Wu, Cary [> >]
Yamada, Yuki [> >]
Zakula, Björn [> >]
Zhang, Nan [> >]
Ziller, Conrad [> >]
Zins, Stefan [> >]
Żółtak, Tomasz [> >]
2021
SocArXiv
No
[en] Immigration ; Meta-Reliability ; Methodology ; Noise ; Policy Preferences ; Political Science ; Political Sociology ; Replication ; Researcher Variability ; Secondary Observer Effect ; Social and Behavioral Sciences ; Sociology
[en] The paper reports findings from a crowdsourced replication. Eighty-four replicator teams attempted to verify results reported in an original study by running the same models with the same data. The replication involved an experimental condition. A “transparent” group received the original study and code, and an “opaque” group received the same underlying study but with only a methods section and description of the regression coefficients without size or significance, and no code. The transparent group mostly verified the original study (95.5%), while the opaque group had less success (89.4%). Qualitative investigation of the replicators’ workflows reveals many causes of non-verification. Two categories of these causes are hypothesized, routine and non-routine. After correcting non-routine errors in the research process to ensure that the results reflect a level of quality that should be present in ‘real-world’ research, the rate of verification was 96.1 in the transparent group and 92.4 in the opaque group. Two conclusions follow: (1) Although high, the verification rate suggests that it would take a minimum of three replicators per study to achieve replication reliability of at least 95 confidence assuming ecological validity in this controlled setting, and (2) like any type of scientific research, replication is prone to errors that derive from routine and undeliberate actions in the research process. The latter suggests that idiosyncratic researcher variability might provide a key to understanding part of the “reliability crisis” in social and behavioral science and is a reminder of the importance of transparent and well documented workflows.
Researchers ; Professionals ; Students
http://hdl.handle.net/10993/47447
10.31235/osf.io/j7qta
https://osf.io/preprints/socarxiv/j7qta/
https://osf.io/preprints/socarxiv/j7qta/

There is no file associated with this reference.

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.