Hide Notes 

Slide 1: Experimental Results of Online Information Aggregation Markets for Sales Forecasting Eric van Heck Rotterdam School of Management Erasmus University evanheck@rsm.nl www.rsm.nl/evanheck INSEAD Presentation Fontainebleau, 20 June 2008 © Eric van Heck, 2008.
Slide 2: Menu 1. What are information aggregation markets (or also called prediction markets)? 2. State-of-the-art in practice IOWA Political Markets Hollywood Stock Exchange 3. State-of-the-art in theory Project 1 with Mathijs van der Vlis 4. State-of-the-art in theory and practice Project 2 with Annie Yang et al. and anonymous company 5. Conclusions
Slide 3: Introduction How many passengers can travel with the Silja Symphony? Helsinki – Stockholm v.v.
Slide 4: Aggregation and Averaging Bo: Heli: Virpi: Ari: Pekka: Mika: Jyrki: Szymon: Ralph: Esko: 2,700 2,640 3,005 3,050 2,502 2,600 2,845 2,777 2,799 3,592 Helsinki – Stockholm v.v.
Slide 5: Aggregation and Averaging Bo: Heli: Virpi: Ari: Pekka: Mika: Jyrki: Szymon: Ralph: Esko: Average: 2,700 2,640 3,005 3,050 2,502 2,600 2,845 2,777 2,799 3,592 2,851 Helsinki – Stockholm v.v.
Slide 6: Aggregation and Averaging Bo: Heli: Virpi: Ari: Pekka: Mika: Jyrki: Szymon: Ralph: Esko: Average: Correct answer: 2,700 2,640 3,005 3,050 2,502 2,600 2,845 2,777 2,799 3,592 2,851 2,852 Helsinki – Stockholm v.v.
Slide 7: Aggregation and Averaging Bo: Heli: Virpi: Ari: Pekka: Mika: Jyrki: Szymon: Ralph: Esko: 2,700 2,640 3,005 3,050 2,502 2,600 2,845 2,777 2,799 3,592 Helsinki – Stockholm v.v. Average: 2,851 Correct answer: 2,852 The average is a very good predictor – wisdom of crowds. Jyrki is closest to the correct answer!
Slide 8: What are information markets? 1. A group of people that buy and sell stocks. 2. Stocks represent the potential outcome of the subject to be forecasted (number of Silja passengers, future demand of mobile telephones, winner soccer game, etc). 3. Market mechanism is a double auction. 4. Market price of a particular stock represent the probability that that potential outcome will happen – for example: stock Italy (in the game Italy – NL) is 0,80 cent (range 0 – 100 cents) = probability that Italy wins is 80%. 5. The market aggregates information by the aggregation of the individual beliefs of the players.
Slide 9: State-of-the-art in practice Some applications in practice: IOWA Political Markets Hollywood Stock Exchange Internal Information Markets for example by HP, Google, and external Information Markets such as NewsFutures, Foresight Exchange.
Slide 10: IOWA Political Markets
Slide 14: Lessons Learned (Berg et al, 1996, 2000) • IOWA political markets perform better than polling results • Presidential election markets perform better than (lower profile) congressional, state, or local elections • Markets with more volume near the election perform better • Markets with fewer contracts (i.e. fewer candidates or parties) predict better
Slide 15: Hollywood Stock Exchange
Slide 16: Trading in MovieStocks
Slide 17: Trading in StarBonds
Slide 18: Lessons Learned • Prices of securities in Oscar, Emmy, and Grammy awards correlate well with actual award outcome frequencies, and prices of movie stocks accurately predict real box office results (Pennock, 2001).
Slide 19: Hype Cycle for Emerging Technologies 2006
Slide 20: State-of-the-art in theory Market Characteristics Incentive Mechanism Market Information/Signals Trading Mechanism Contract Type (binary, spread, index) Frequency of information update Liquidity/Market Size Selling short/portfolios Information Cascades/Market Bubbles Market Efficiency Trader Anonymity Transaction Costs Prediction Metric (last trading price, avg price) Trader Characteristics TraderType Biases/Bounded Rationality Information Source Trading Experience/Knowledge Private Information Cheating/Collaboration/Manipulation Trader Demographics Homogeneity/Heterogeneity Wealth Risk Attitude Characteristics of the tobe-predicted event Inherent Predictability Aggregate Certainty Information Availability/Costs Time Scope
Slide 21: Main Theories • Mechanism Design Theory (Hayek 1945) Markets are an appropriate mechanism for the purpose of efficient information aggregation and decision making due to the incentives for information discovery. Double Auction Theory (Plott and Sunder, 1982, 1988) Prediction markets have the ability to aggregate dispersed private information held by individuals as the double auction mechanism has the ability to disseminate private information among traders. Rational Expectation Theory (Lucas 1972, Grossman 1981) The price observed in a prediction market is a sufficient statistic for all information available to traders The Wisdom of Crowds (Surowiecki 2004) Small and large groups of people seem to do better at decision making than individuals. • • •
Slide 22: Mechanism Design Theory
Slide 23: Project 1 - with Mathijs van der Vlis : What is the impact of the number of traders, the distribution of wisdom, and monetary incentives to the outcome of information markets?
Slide 24: Hypotheses 1. Number of traders (Surowiecki, 2004) More traders will increase the level of aggregation and the level of prediction accuracy 2. Distribution of wisdom (Anderson and Holt, 1997; Hansen, Smith, and Strober, 2001; Hanson and Oprea, 2004) Uneven distribution among traders will increase the level of aggregation and the level of prediction accuracy 3. Monetary incentives (Servan-Schreiber, Wolfers, Pennock, and Galebach, 2004) Monetary incentives will not increase the level of aggregation and the level of prediction accuracy
Slide 25: 128 Laboratory Experiments to forecast future demand of mobile telephones
Slide 26: Results Experiments (N = 128)
Slide 27: Hypotheses 1. Number of traders (Surowiecki, 2004) More traders will increase the level of aggregation and level of prediction accuracy No Yes 2. Distribution of wisdom (Anderson and Holt, 1997; Hansen, Smith, and Strober, 2001; Hanson and Oprea, 2004) Yes No Uneven distribution among traders will increase the level of aggregation and the level of prediction accuracy 3. Monetary incentives (Servan-Schreiber, Wolfers, Pennock, and Galebach, 2004) Yes Yes Monetary incentives will not increase the level of aggregation and the level of prediction accuracy
Slide 28: Lessons Learned • Results indicate that even in the presence of a small number of traders there tends to be aggregation, while only in the presence of a large number of traders are accurate predictions generated. • When wisdom is unequally distributed there is aggregation (wise people lead markets), yet the markets do not produce more accurate predictions (wise people can potentially mislead markets). • Monetary incentives impact neither the level of aggregation nor the level of accuracy.
Slide 29: State-of-the-art in theory and practice Some applications: Helsinki – Stockholm v.v. Internal Information Markets for example at a financial company
Slide 30: Project 2 - with Annie Yang, Maarten Colijn, Willem Verbeke, Mathijs van der Vlis and anonymous company What is the performance of information markets in forecasting the overall sales of a product over several regions in the Netherlands?
Slide 31: Hypotheses • Market Size – Number of Traders (Surowiecki 2004, Hansen 2003) H1a: A prediction market with more traders is likely to aggregate sooner and more significantly. H1b: A prediction market with more traders is likely to forecast more accurately. • Monetary Incentives (Servan-Schreiber et al. 2004) H2: An offer of monetary incentives does not affect the activeness of traders’ participation in a prediction market. • Time Horizon (Berg et al. 2003) H3: A prediction market forecasts more accurately in a short run than in a long run.
Slide 32: Trading Web Page
Slide 33: 1st Prediction Market Subject to be predicted Contracts Annual sales of a financial product Spread contracts (in million euro) Regional sales managers 10 34 31 8 604 2nd Prediction Market Periodical sales of a financial product Spread contracts (in million euro) Regional sales managers 9 34 18 3 461 Description of Prediction Markets Description of traders Number of stocks Number of traders Number of active traders Number of very active traders Total number of bids (incl. demand and sell) Total number of completed bids (buy and sell) Time of markets Market duration 368 275 24 hrs / 7 days 12 calendar days (Feb 2007) 24 hrs / 7 days 12 calendar days (June 2007)
Slide 34: Aggregation and Forecasting Results Historical Stock Prices in 1st Prediction Market 80 110-120 Actual sales i.e. 133 Market forecast Top-down forecast 70 121-130 131-140 141-150 Stock Price (in point) 60 50 40 30 20 10 0 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 151-160 161-170 171-180 181-190 191-200 201-210 Trade Trading Day
Slide 35: Aggregation and Forecasting Results Historical Stock Prices in 2nd Prediction Market 120 19 - 22 Market forecast Top-down forecast Actual sales i.e. 28.6 100 22 - 25 25 - 28 Stock Price (in point) 80 60 40 20 0 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 28 - 31 31 - 34 34 - 37 37 - 40 40 - 43 43 - 46 Trade Day Trading Day
Slide 36: Forecasting Accuracy 1st Prediction Market % error 2nd Prediction Market % error Actual results 133 28.6 Prediction market forecast 141-150 +6% - 13% 19-22 -23% - 34% Top-down forecast 150-160 +13% - 20% 27 -6%
Slide 37: Hypotheses • Market Size – Number of Traders (Surowiecki 2004, Hansen 2003) H1a: A prediction market with more traders is likely to No aggregate sooner and more significantly. H1b: A prediction market with more traders is likely to forecast Yes more accurately. • Monetary Incentives (Servan-Schreiber et al. 2004) H2: An offer of monetary incentives does not affect the Yes activeness of traders’ participation in a prediction market. • Time Horizon (Berg et al. 2003) H3: A prediction market forecasts more accurately in a short run than in a long run. No
Slide 38: Lessons Learned 1. Market size, in terms of the number of traders, does not necessarily influence market aggregations but the accuracy of predictions. A thicker market is more likely to forecast accurately. 2. Monetary incentives are not effective to motivate traders to trade in internal prediction markets – time for trading is a constraint. 3. Markets predict more accurately in a long run than in a short run. Interesting because the impact of the worldwide mortgage crises was predicted very well 4. Traders are sensitive to the prices of contracts, learning from signals and constantly updating their beliefs. However, this yields that traders could be easily misled, particularly in a thin market.
Slide 39: Conclusions 1. “Information Aggregation” is a Key Critical Component for Firms - online markets can improve the information aggregation capability of a firm! 2. Several issues need to be solved for example: details of the market design incentive structure of players 3. Do you want to know more: please join! Helsinki – Stockholm v.v.