The Market Scoring Rule
Of course, scoring rules themselves must be compared against other means of eliciting forecasts. Whereas the Delphi Method seeks to maximize information-sharing and collaboration, a scoring rule seeks to elicit the best possible forecast from a single forecaster. Prediction markets, we have seen, generally seek to obtain consensus forecasts rather than simple individual forecasts. Robin Hanson has proposed a prediction market design that builds on the concept of scoring rules He calls this design the market scoring rule.52 Hanson’s approach is simple. The sponsor of the market scoring rule makes an initial prediction of the relevant probability or probabilities. Others are then permitted to offer replacement predictions, under one condition: any predictor after the first must compensate the prior predictor according to the specific scoring rule, for example, a logarithmic scoring rule. The last predictor is compensated by the market sponsor according to this scoring rule. The insight is that each successive predictor will expect to receive money for improving on the prior prediction (or pay money if the predictor in fact makes a worse prediction). Payouts for marginal improvements in predictions are thus precisely as indicated in figure 4.1. Because it is often desirable to provide constant subsidies across the probability spectrum, the quadratic market scoring rule will be the best choice in many contexts.53 One virtue of the market scoring rule is that it can be easily harnessed to make predictions of numbers other than probabilities. Suppose, for example, that a market sponsor is interested in predicting next year’s Even when used to assess probabilities, the market scoring rule has significant virtues relative to a probability estimate prediction market. Unlike such a market, the market scoring rule should work well in situations of low liquidity. Like the dynamic pari-mutuel market, the market scoring rule can be presented to a user just as a probability estimate prediction market would be. That is, the user can be presented with bid and ask queues, but the entries on those queues are made by the market sponsor rather than other traders. With this approach, the user can purchase tradable contracts that will pay a set amount if a particular event occurs. The market scoring rule functions as an automated market maker that will always accept buy or sell orders at the price corresponding to current market probabilities. It will buy or sell only an infinitesimally small share at that price, however, and so for any given scoring rule, calculus must be used to derive the equations that can be used to calculate the number of shares that can be purchased for a given amount of money or to move the consensus probability estimate to a particular number. As Hanson points out, a market scoring rule should provide appropriate incentives when it turns out that only one individual decides to make a prediction. In that case, the market scoring rule amounts to a scoring rule. Unlike a traditional scoring rule, however, the market scoring rule provides incentives for individuals to make new predictions when they believe that the initial predictor might have made a poor prediction, for example, if there is suspicion that the initial predictor might be trying to manipulate the market. The knowledge that this might occur will decrease the initial predictor’s willingness to attempt manipulation. In order to succeed in manipulating the market scoring rule, a participant would need to be willing to repeatedly revert the prediction against determined challengers back to the value that the participant prefers. In effect, to manipulate the market, one must be willing to take bets against anyone who might challenge the participant and to do so repeatedly. A series of back-and-forth predictions between two competing traders will likely attract interest in the market and independent third-party evaluations. So a manipulator can succeed only if he or she has more money to spend than the rest of the potential market combined or if virtually no one else in the market notices and decides to challenge the manipulator’s evaluation. Because of its particular applicability to low-liquidity environments, a primary competitor of the market scoring rule is the dynamic pari-mutuel market. With both the market scoring rule and a dynamic pari-mutuel market, a participant–it might be inaccurate to call the participant a trader–can profit on information without finding someone with an opposite prediction to enter into a transaction. A significant difference between the two approaches is in incentives related to the timing of trading. We saw in Chapter 3 that with the dynamic pari-mutuel market, a trader with private information sometimes might have an incentive to wait before betting on the information, because a shift in the market will have a bigger payout when more money has been invested in the payout. This is a particular danger if others change their probability evaluations based on the trader’s bets. This danger does not occur with the probability estimate prediction market, because there is no inherent trend toward increasing liquidity over the course of the market. With the market scoring rule, meanwhile, there will ordinarily be an incentive to invest on the basis of information as soon as possible, because one earns money only to the extent that one improves on the prior prediction. Another potential advantage of the market scoring rule is that it provides a relatively straightforward method of subsidizing the market.54 The market sponsor needs simply to identify a scoring rule that converts the success of the final predictor into dollars. This approach, however, shares two disadvantages of the “seed wager” approach to subsidizing the dynamic pari-mutuel market. First, the subsidy cannot be fixed in advance. Where a market sponsor runs a large number of markets, a solution is for the market sponsor to announce a fixed subsidy for all of the markets. Each individual market would then be run only with points, and the conversion between points and dollars would be determined after the close of all markets, set at the level that will precisely deplete the subsidy. Second, a great proportion of the subsidy might be awarded to the first market participant to make an announcement, especially if the initial estimate offered by the market sponsor is weak. A possible partial solution is to hold an auction for the right to be the initial predictor, with any revenues from the auction added to the amount to be paid to the final predictor in the market. According to this approach, the market sponsor at least will not be wasting a subsidy on information that is publicly available. An alternative approach might be to alter the subsidy level over time. The market scoring rule might be more appropriate in circumstances in which the underlying events being predicted are highly volatile, and the dynamic pari-mutuel market might be superior where there is relatively little underlying volatility over the course of the market. The dynamic pari-mutuel market will tend to be less sensitive to new predictions relatively late in the betting cycle, but bets that are large enough can always push the prediction any desired distance. High volatility will mean that for each successive probability shift, increasingly large amounts of money will need to be invested in the market to move the market to the new consensus probability, and increasingly large amounts of money can be earned by fixing the probability estimates. A consequence is that the dynamic pari-mutuel market will provide greater liquidity toward the end of the market. With the market scoring rule, by contrast, it will always cost the same amount to move the probability prediction from a particular point to another, regardless of how much has been invested in the market. The decreased sensitivity of the dynamic pari-mutuel market to bets of a fixed size might be seen as a virtue, however, especially in low-volatility environments. The possibility that a single announcement could dramatically change the market price might be a concern with the market scoring rule. At least participants in a prediction exercise based on the market scoring rule need to establish that they will be able to pay the appropriate amount if a prediction turns out to be a bad one. If participants were not required to pay off the previous predictor in advance or to show evidence of ability to pay through the market sponsor later on, there might be an incentive to change the market prediction arbitrarily. The final consideration in the comparison between the market scoring rule and other approaches is that the market scoring rule, like scoring rules, may provide differing levels of incentive to gather information and improve the market prediction at different portions of the probability spectrum, as illustrated in figure 4.1. In addition, as figure 4.2 shows, with various versions of the market scoring rule, the reward for moving the probability estimate closer to the correct value decreases as the market value approaches this value. As in figure 4.1, payouts are normalized and expressed as a proportion of the increased payout for an improvement in probability assessment from 0.01 to 0.02. In figure 4.2, however, the correct probability is 0.5, but making the last incremental shift from 0.49 to 0.50 produces almost no profit. These problems might be more of a concern with the market scoring rule than with traditional scoring rules because a major purpose of market scoring rules is to give participants incentives to make small changes in evaluations, but this may be difficult to achieve for many portions of the probability spectrum. In many cases, however, there is a potential solution: sequential use of two different market scoring rules. The key insight is that one prediction market can be used to forecast the result of another. As long as the second prediction market is structured in a way that guarantees that forecasters will have an incentive to be honest, the first prediction market can be structured in a way that ensures that the marginal expected profit from improving a probability prediction will remain constant across the probability continuum. This approach would work as follows: The second process would use the quadratic market scoring rule or one of the other strictly proper scoring rules discussed above, but the first process would use a simple linear scoring rule and would predict the prediction of the second scoring rule, rather than the actual event. Suppose, for example, that what is being predicted is a two-horse race in which the first horse has a 75 percent chance of winning. The race will directly affect profits of participants in the second process, but not in the first. As we have seen, a linear scoring rule ordinarily would give a forecaster an incentive to announce a prediction that the first horse will win with certainty, but in this case a forecaster would recognize that the second market scoring rule will end with a prediction of 0.75, and so it is better to announce a prediction of 0.75 in the first market scoring rule period. The second market scoring rule could occur over a very short period of time, potentially with quite a modest subsidy, and in general it should not change the prediction of the first market scoring rule by much. The purpose of the second market scoring rule would only be to discipline the first. The technique should work as long as the second market can be completed before the conclusion of the second event becomes obvious. It might not work as well in cases in which an outcome (rather than a probability) becomes increasingly obvious as the relevant event approaches, as for example with predictions of the weather.
Leave a Reply