#MTRP: Mechanical Turk Replication Project

inclusion criteria

How were hypotheses to be tested chosen? For each study we have selected one statistically significant within-subject or between-subjects treatment effect or interaction effect for replication, using the following criteria:

  • Randomly pick a MTurk experiment reporting a statistically significant (p < 0.05) treatment effect or an interaction with a treatment effect for papers reporting more than one MTurk experiment.
  • Include the most central statistically significant result of this study/experiment (as highlighted in the original study).
  • If more than one equally central result was reported, randomly pick one of the results for replication.
  • If an original study included more treatments than the treatments selected for replication, we focus on replicating the treatments used for the result selected for replication.
  • We will use the same statistical test in the replication as in the original study.

studies to replicate

The table below summarizes the 41 studies selected to be included in the #MTRP (based on the inclusion criteria sketched above). Note that the table columns are sortable: clicking on a column header once will sort the table (based on the particular column's values) in ascending order; clicking on the header twice will sort the table in descending order. Hovering over the -symbol in the "Hyp." column, will show a pop-up window containing the paper's title and the hypothesis to replciate and bet on. The -symobl in the right-most column will direct you to the replication report (in PDF format), outlining all relevant details.
# Authors Year Hyp. Interaction
Effect		 Within-
Subject Test		 Orig.
Cohen's d		 Orig.
p-Value		 Orig.
Sample		 Rep.
Sample		 Replication
Report
1 Ames & Fiske 2015 no no 0.3651 0.0096 201 710
2 Atir & Ferguson 2018 no yes 0.0912 0.0322 554 2842
3 Baldwin & Lammers 2016 yes no 0.6279 0.0276 200 960
4 Bear et al. 2017 yes no 0.1865 0.0033 994 2719
5 Boswell et al. 2018 no no 0.6662 <0.0001 260 260
6 Caruso et al. 2016 no no 0.3246 0.0011 410 898
7 Casella et al. 2017 no no 0.2839 0.0006 596 1174
8 Chao 2017 no no 0.2043 0.0204 519 2267
9 Cheon & Hong 2017 no no 0.3204 0.0400 167 922
10 Clarkson et al. 2015 yes no 1.1189 0.0015 135 303
11 Cooney et al. 2016 yes yes 0.6722 <0.0001 120 120
12 Côte et al. 2015 yes no 0.3241 0.0319 704 3601
13 Flesch et al. 2018 no no 0.4761 0.0225 95 418
14 Genschow et al. 2017 no no 0.1871 0.0362 504 2702
15 Gheorghiu et al. 2017 no yes 0.1065 0.0321 408 2085
16 Guilbeault et al. 2017 no no 1.3093 0.0013 24 56
17 Halevy & Halali 2015 no no 0.9011 <0.0001 198 198
18 Handley et al. 2015 yes no 0.4596 0.0464 303 1791
19 Hoffman et al. 2016 yes no 0.8708 0.0398 92 499
20 Hofstetter et al. 2017 no no 0.3472 0.0020 323 785
21 Horne et al. 2015 no no 0.3366 0.0168 205 835
22 Isley et al. 2016 no yes 0.1621 0.0023 355 901
23 Jachimowicz et al. 2017 no no 0.7076 0.0071 62 189
24 John et al. 2016 no no 0.3491 0.0393 142 776
25 Jordan et al. 2016 no no 0.2298 0.0018 735 1791
26 Jun et al. 2017 no no 0.3688 0.0114 192 696
27 KC et al. 2018 no no 0.3625 0.0189 171 720
28 Klein & O’Brien 2018 no no 1.4735 <0.0001 207 207
29 Kouchaki & Gino 2016 no no 0.4031 0.0012 258 583
30 Kraus et al. 2017 no yes 0.4943 <0.0001 202 202
31 McCall et al. 2017 no no 0.4228 <0.0001 480 529
32 Morris et al. 2017 yes no 0.9859 <0.0001 100 100
33 Mummolo 2018 no no 0.1658 0.0205 784 3440
34 Payne et al. 2017 no no 0.2973 0.0281 221 1070
35 Phillips & Cushman 2017 no no 0.5057 <0.0001 460 460
36 Rai et al. 2017 no no 0.3350 0.0250 182 843
37 Reeck et al. 2017 no no 0.3070 0.0272 207 1004
38 Schilke et al. 2015 no no 0.4202 <0.0001 413 536
39 Stern et al. 2015 no yes 0.2209 0.0003 273 485
40 Vacharkulksemsuk et al. 2016 no no 1.7461 <0.0001 426 426
41 Williams et al. 2016 no no 0.9512 <0.0001 96 105