We will measure peer beliefs about the replicability of the 41 studies included in our project by running incentivized decision markets with a sample of social scientists acting as traders. These decision markets are similar to prediction markets in many ways - we ask the participants to give their assessment of the likely outcome of the replication process. Similar to previous studies (Dreber et al., 2015; Camerer et al., 2016, 2018), researchers will be provided with monetary endowments and will trade on the outcome of the hypotheses via an online market platform. We will run a market for each hypothesis (i.e., each study), predicting whether or not the hypothesis will replicate in the replication experiment. A decision rule is then used to select which studies to actually replicate based on the predicted outcomes. Importantly, the decision rule is stochastic, with each study having a strictly positive probability of being picked for replication. Payoffs are only given for predictions regarding the outcomes of the selected studies, but they are scaled up by the inverse of their probability in the decision rule (Chen et al., 2014). Thus, while prediction markets pay 1 Token per share on the correct outcome, decision markets pay 1 Token divided by the probability in the decision rule.
Our decision rule is the following: we will replicate the 12 studies with the lowest market prices, the 12 studies with the highest market prices, and 2 random studies out of the remaining studies. This ensures that participants’ expected payoffs in decision markets are analogous to those made in properly incentivized prediction markets, and the game-theoretical results on strategic interactions between participants in prediction markets carry over.
The markets will open for trading on Monday, November 8, 2021, 4pm UTC (coordinated universal time) and close on Monday, November 22, 2021, 4pm UTC.
In case you have any questions regarding the prediction markets, please contact Felix Holzmeister via mturk-replications@uibk.ac.at.
We cordially invite you to participate!
How can I sign-up for the decision markets? Registrations to participate in the decision markets are administered via the sign-up form at the bottom of the page. A PhD degree (or currently being a PhD student) or a comparable level of expertise in the social sciences is required for participating to ensure expertise in the field. Registration to participate in the markets will close on Friday, October 29, 2021 at 4pm UTC.
Who can and cannot trade? Beforing entering the markets, we will ask you to fill out a survey asking for your replication beliefs about the hypotheses included in the decision markets. The link to the survey will be sent to you via e-mail on October 8, 2021, or a few days after your registration (in case you sign up after October 8). Completing the survey by Friday, November 5, 2021 will be a prerequisite to participate as a market trader.
All replicator staff – faculty, research assistants, and technical staff who are conducting the replications – will not be eligible to trade in the markets, and are expected to keep themselves as uninformed as possible about activity in the markets until the replications have been finished. The only team member who will have access to the decision market data while markets are up an running is Felix Holzmeister; all other contributors will be blinded to the activities on the market.
How do decision markets work? For each hypothesis, the decision market price serves as an aggregate measure of peer beliefs about the replicability of the corresponding hypothesis. Detailed information on the market mechanism and instructions on how to use the software are summarized below.
What does it mean for a study to replicate? The decision market requires clear-cut criteria to define the outcomes of the replication process. We define a result as being successfully replicated if the statistical method used in the original study yields a two-sided p-value < 0.05, and the effect is in the same direction as in the original study.
How can I find out details of the replication procedure and test statistics? Our approach is to make every possible effort to replicate all aspects of the design, software, and analysis as closely as possible. To make the details of the replications transparent, brief “replication reports” are available to market participants which describe the hypotheses and relevant differences between the original design and the planned replication. For more information on the replicated studies and the hypotheses, please refer to the studies tab.
How can I access the decision markets? The market can be accessed via a dedicated trading platform developed for the project. The software is available via markets.mtrp.info. Trader ID’s and passwords will be sent in a separate e-mail.
How will personal data be handled? When analyzing data from the market, we will identify participants solely through their trader ID. Personal data such as e-mail addresses or names will not be used in the analysis, will be removed from any published data, and will be deleted once payments for the decision markets have been made.
How will market performance be incentivized? Trading on the decision markets will be incentivized, i.e., traders will be paid based on their performance in the markets. Payments will be made after the markets close and the asset payoffs are determined by the actual replication outcomes. When signing up for the markets, participants have the opportunity to indicate whether the final payment shall be paid via Amazon gift cards (either for amazon.com, amazon.co.uk, or amazon.de) or PayPal.
You can trade on the outcome of each of the 41 hypotheses in a web-based market interface. Hence, we run one market for each hypothesis. Detailed information about the hypotheses, the experimental protocol, and the data set are available via studies tab.
Your initial endowment is 100 Tokens (= $50), the market’s experimental currency unit. You can trade on as many assets, i.e., hypotheses, as you want to. For each asset, you can trade shares to forecast whether or not the study will be replicated. Hence, the market price is the aggregate belief of traders for the likelihood that the replication experiment will result in an effect in the same direction as the original study with p < 0.05. Prices can fluctuate between 0 and 1.
The number of shares you hold when the market closes determines the payoff you will receive. This number can be positive or negative. If you hold a positive number of shares and the corresponding study is selected for replication, you will receive a payoff if the replication is successful. If you hold a negative number of shares and the corresponding study is selected for replication, you will receive a payoff if the replication is not successful.
Once the replications have been conducted, the true “fundamental value” (FV) for each asset (i.e., the binary outcome whether the replication experiment yields an effect in the same direction as the original study with p < 0.05) is revealed and gains and losses are calculated as follows. Please note that you will only receive a payout for the shares you hold. Any Tokens not invested into shares when the market closes will be voided.
Once you have identified a market that you would like to trade on, click on the corresponding button to access it. In the trading interface, you will face two possibilities to trade:
The trading takes place via a computerized market maker using a logarithmic market scoring rule following Hanson (2007). When you invest Tokens to buy shares (i.e., if you increase a position), the price of these shares will move up. When you sell shares (i.e., if you decrease a position), the price of these shares moves down.
Additional information on the pricing mechanismYou can track all your positions in the market overview when logged-in to the market software. For each asset, you can see the current prices (“Price”), how many shares you hold in a particular asset (“Shares Held”), and the current value of your position in Tokens (“Current Value”). The current value of a position is calculated as the number of Tokens you would receive by immediately closing your position in the asset.
Additional information on the value of your assetsAll Tokens which have not been invested will be voided when the market closes. All assets will be settled once all replication experiments have been completed. Each asset has a clear event outcome determining its fundamental value (FV). The 12 studies with the highest market price, and the 12 studies with the lowest market price will definitely be selected for replication. For shares on these studies, if the hypothesis is successfully replicated, you receive 1 Token (corresponding to $.50) for each positive number of shares you hold, but nothing for negative share holdings. If the hypothesis is not successfully replicated, you receive 1 Token (corresponding to $.50) for each negative number of shares you hold, but nothing for positive share holdings. From the remaining 17 studies, two additional studies will be randomly selected for replication. For shares on these studies, if the hypothesis is successfully replicated, you receive 8.5 Tokens (corresponding to $4.25) for each positive number of shares you hold, but nothing for negative share holdings. If the hypothesis is not successfully replicated, you receive 8.5 Tokens (corresponding to $4.25) for each negative number of shares you hold, but nothing for positive share holdings. The remaining 15 studies will not be selected for publication. You will not receive any payouts from holdings on these studies, regardless of your holdings being positive or negative.
Payouts are capped at $500 per subject and will be paid out in March 2021. You will have the opportunity to choose between receiving your payment as an Amazon gift card (amazon.com, amazon.co.uk, or amazon.de) or via PayPal when signing-up as a market participant.
Additional information on paymentsThe payout of a position in a certain asset depends on two numbers: the terminal value of the asset, i.e., its fundamental value FV and the number of shares you hold.
Example: Assume you hold 50 shares of hypothesis A, 30 shares for hypothesis B, -20 shares for hypothesis C, 10 shares for hypothesis D, and you did not trade for hypothesis E. Hypothesis A and hypothesis B were among the 12 studies with the highest market. Because of this, both were selected for replication. Hypothesis A was successfully replicated, Hypothesis B was not successfully replicated. You would receive 50 Tokens (50 shares x 1 Token) for your holdings in hypothesis A, and 0 Tokens (30 shares x 0 Tokens) for your holdings in hypothesis B. Hypothesis C and Hypothesis D were not among the 12 hypotheses with highest or lowest market price. Hypothesis C was among the two randomly sampled hypotheses, and did not successfully replicate. Hypothesis D was not among the randomly sampled hypotheses. You will receive 170 Tokens (20 shares x 8.5 Tokens) for your holdings in Hypothesis C, and 0 Tokens for your holdings in Hypothesis D. Hypothesis E was among the 12 studies with lowest market price. It was selected for replication, and did not replicate. Since you have no holdings, you will not receive any payoff for this hypothesis. Your final payment, thus, would be 50 + 0 + 170 + 0 = 220 Tokens. These will be valued at a rate of 1 Token = $0.50. Hence, you would receive a payment of $110 via your preferred payment mechanism.Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science 351(6280): 1433-1436.
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Kichler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., Isaksson, S., Manfredi, D., Rose, J., Wagenmakers, E.-J., & Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2(9): 637-644.
Chen, Y., Kash, I. A., Ruberry, M., & Shnayder, V. (2014). Eliciting Predictions and Recommendations for Decision Making. ACM Transactions on Economics and Computation 2(2): 6:1-6:27.
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015). Using Prediction Markets to Estimate the Reproducibility of Scientific Research. Proceedings of the National Academy of Sciences 112: 15343-15347.
Hanson, R. (2007). Logarithmic market scoring rules for modular combinatorial information aggregation. Journal of Prediction Markets 1(1): 3–15.