#MTRP: Mechanical Turk Replication Project

What exactly are we doing? We have selected 41 experimental studies in the social sciences conducted on MTurk published in the Proceedings of the National Academy of Science (PNAS) in 2015-2018. We will replicate 26 of these, and all studies have a positive probability of being chosen for replication. The 41 papers were objectively chosen because they were published in this high-profile journal in this time period, they are online studies involving subjects from Amazon Mechanical Turk, they share a common structure in testing a treatment or interaction effect within or between subjects, and they test at least one clear hypothesis with a statistically significant finding. We plan to conduct the replications starting in January 2022 and throughout the spring of 2022.

Replication procedure: We will carry out the data collection based on having 90% power to detect 67% of the effect size reported in the original study (with the effect size transformed into the Cohen's d, to have a common standardized effect size measure across all studies). The criteria for successful replication is an effect in the same direction as the original study and a p-value < 0.05 (in a two-sided test). This is the replication outcome traders are betting on - but in the paper we will also include other replication criteria.

Choosing what to replicate: We will run a decision market where we will ask participants to help us choose which studies to replicate. All studies will have a positive probability of being chosen for replication. Please see the tab “Decision Markets” for more information. In short, decision markets function similarly to prediction markets in that traders bet on the replication outcome, but not all studies will actually be replicated. The selection of the 41 studies does not reflect any subjective belief of ours about whether or not these studies were especially likely to replicate.