Questions continue to be raised about the accuracy of the polls. Obviously, in just a few more days, we’ll know which polls were right (on average) and which were wrong. But in the meantime, it’s useful to understand how the polls are – at the very least – different from one another, and form a baseline set of expectations to which we can compare the election results on Tuesday. The reason this question takes on special urgency now is that there’s essentially no time left in the campaign for preferences to change any further: if the state polls are right, then Obama is almost certain to be reelected.
In previous posts, I’ve looked at both house effects, and error distributions (twice!), but I want to return to this one more time, because it gets to the heart of the debate between right-leaning and left-leaning commentators over the trustworthiness of the polls.
A relatively small number of survey firms have conducted a majority of the state polls, and therefore have a larger influence on the trends and forecasts generated by my model. Nobody disputes that there have been evident, systematic differences in the results of these major firms: some leaning more pro-Romney, others leaning more pro-Obama. As I said at the outset, we’ll know on Election Day who’s right and wrong.
But here’s a simple test. There have been hundreds of smaller organizations who have released fewer than a half-dozen polls each. Most have only released a single poll. We can’t reliably estimate the house effects for all of these firms individually. However, we can probably safely assume that in aggregate they aren’t all ideologically in sync – so that whatever biases they have will all cancel out when pooled together. We can then compare the overall error distribution of the smaller firms’ surveys to the error distributions of the larger firms’ surveys. (The survey error is simply the difference between the proportion supporting Obama in a poll, and my model’s estimate of the “true” proportion on that state and day.)
If the smaller firms’ errors are distributed around zero, then the left-leaning firms are probably actually left-leaning, and the right-leaning firms are probably actually right-leaning, and this means that they’ll safely cancel each other out in my results, too. On the other hand, if the smaller firms’ error distribution matches either the left-leaning or the right-leaning firms’ error distribution, then it’s more likely the case that those firms aren’t significantly biased after all, and it’s the other side’s polls that are missing the mark.
What do we find? This set of kernel density plots (smoothed histograms) shows the distribution of survey errors among the seven largest survey organizations, and in grey, the distribution of errors among the set of smaller firms. The smaller firms’ error distribution matches that of Quinnipiac, SurveyUSA, YouGov, and PPP. The right-leaning firms – Rasmussen, Gravis Marketing, and ARG – are clearly set apart on the pro-Romney side of the plot.
If, on Election Day, the presidential polls by Quinnipiac, SurveyUSA, YouGov, and PPP prove to be accurate, then the polls by Rasmussen, Gravis Marketing, and ARG will all have been underestimating Obama’s level of support by 1.5% consistently, throughout the campaign. Right now, assuming zero overall bias, Florida is 50-50. The share of Florida polls conducted by Rasmussen, Gravis Marketing, and ARG? 20%. Remove those polls from the dataset, and Obama’s standing improves.
Four days to go.
Dr. Linzer (since degrees were so in evidence in the last thread), thanks for this analysis. This is a fresh, data-based perspective on something I have suspected, i.e., that certain pollsters may be using their polling in an attempt to establish a narrative favorable to those pollsters’ political leanings, much the same way that campaigns have long released internal polls to drive a narrative favorable to the campaign.
It appears that Gresham’s law now applies to political polling, and the value of polling is being debased by the frequency of robopolling by certain pollsters.
I have made a mental note to swivel back after the election and check how well each of the following groups performed;
1. The media pundits – my feeling is that the urge for supportive propaganda has gotten the better of both left and right. They also have a vested interest in keeping the election close as it boosts their relevance.
2. The polling companies – in a cynical world I fear that many of the polling groups have become tainted by who funds their research. It’s simply not good enough to make one good prediction directly before the election and claim that a last minute event caused a major swing. Sandy?
3. The political scientist / statisticians – I suspect these folks will be the closest to being correct and I believe we owe these smart minds a debt of gratitude For “trying” to call an honest picture. However, I’m wary of next time around when this relatively new “science” may well become another untrustworthy source.
Nice trick for (potentially) factoring out bias.
Drew, great creative analysis! enjoying the blog.
Its what Dr. Wang said too.
At his temporary site.
http://election.princeton.edu/romentum-rove-1nov2012.php
“For a bell-shaped curve, the average error is supposed to be 0.8 sigma. Here it’s much larger, 1.9 sigma. Aha…here may be our culprit. Evidently, national polls have systematic problems. Answer: national polls do about 2.5x worse at predicting the popular vote outcome than expected if the wisdom of crowds of pollsters were perfect.”
Do you think Nate Silver and Mark Blumenthal will agree? I put the link to Dr. Wang’s post in their comments, and tweeted it to Sean Trende.
For what its worth, Dr. Linzer, I agree also.
good work!
I have a couple of questions to ask you.
1. How many polls did you use to produce the error distribution of the smaller firms? In the article you mentioned that “hundreds” of smaller firms conducted surveys, so did you really use that many polls here?
2. What were the composition of the polls you used by the smaller firms? Were they all state polls? If so, were they all surveys from the swing states or were other states represented? Can you give a list of all the polls you used and show which states they came from?
3. Jay Cost wrote an article a couple of days ago showing the error distribution of polls in Ohio (or at least the polls of the major polling firms in Ohio); and while the error distribution certainly isn’t bimodal, I agree that it is somewhat non-normal, as he puts it. (He also points out that the average of the polls gives Obama a 1.8% lead, while the median has Obama up by just 1.) I’ve seen the graphs you posted showing the error distribution of the polls in all the swing states, and besides Ohio (and maybe Colorado) they all seem normal. I’m extremely skeptical about the possibility that only the polls in Ohio are showing a systematic Obama bias, but do you think Jay Cost’s argument has any merit?
I have been telling Nate Silver for two months that he needed to use nonparametrics to correctly estimate house effect because of asymmetrical political behavior.
/vindicated
uberhikari: 1. I start looking at the polls May 1; the total number by “smaller firms” in the plot was about 200. 2. Yes, state polls only, all states. 3. Yes, you’ll get overdispersed (wider than normal) error distributions as a result of random sampling variation plus minor house effects, which is what it looks like we’re seeing.
uberhikari, what Dr. Wang and Dr. Linzer are saying is that (i think) between group differences in the nat’l vs state polling are attributable to systematic methodology errors.
Cost is looking at within group differences in Ohio polling. Not at all the same thing.
Oh, this is brilliant.
https://twitter.com/DrewLinzer/status/264342482379214850
bravo
Nice! I do have one concern about the initial assumption of bias being averaged out across the smaller polling firms. Might they possess, rather than a systematic ideological bias, a more prosaic bias that achieves the same effect? Perhaps the majority of these firms buy their “polling software” (if such a thing exists) from the same vendor, or maybe smaller firms tend to gravitate towards common (cost saving?) techniques that skew their results in the same direction as the ‘pro-Obama’ polling outfits, without sharing a common ideology? Or, as ever, perhaps the answer lies somewhere in the middle. Time will tell!
Drew –
This is useful. So you are saying that we need to calibrate the zero (in error) by using these small firm polls.
DK if you do this but it would be nice if you could control for item level parameters (how the question is asked, DK offered or not) etc. and even any consistent question order stuff (hard so ignore for now) and then estimate house effects. Some house effects may be ‘explained’ by such things.
Ok, there is clearly a difference between the majority of polls, and the right-leaning polls. But not sure that assuming that the group of smaller polls is “correct” is the proper thing to do.
Rasmussen has an interesting history: in 2004, they did VERY well. They hit almost every swing state final margin by 2 points or less, and there was almost no bias, even Slate said they nailed it, and many of their competitors did not.
FL +2.1 dem
OR +1.7 dem
WI +1.5 dem
IA +1.1 dem
MI +0.9 dem
NV +0.7 dem
PA +0.2 gop
OH +1.5 gop
NM +2.3 gop
MN +2.6 gop
However, in 2008, they missed almost every swing state in the GOP direction only. No decminals for this one, but here’s what their site says:
NV +8 gop
MI +6 gop
WA +6 gop
WI +6 gop
CO +5 gop
PA +5 gop
OH +4 gop
IN +4 gop
ME +4 gop
FL +3 gop
NC +2 gop
VA +2 gop
MN +2 gop
MT +1 gop
MO +1 dem
If Rasmussen’s polling methods are the same, then there’s a *structural* difference between the 2004 and 2008 elections. Some intangible we’re not measuring, maybe enthusiasm or something. So the bigger question is: is this like 2004 or like 2008?
I’m looking at the polls two ways now: (1) looking at aggregators and analyzers like Votamatic and 538 and RCP, and (2) looking at just the Rasmussen polls, assuming they have a 1.5 point GOP bias, just in case this is 2004 all over again. My *inclination* is that the poll averages are overstating the Dem case a little bit, maybe 1.0 points, which means that they’re still the favorite but states like Ohio and Iowa could end up being *really* close, and subject to shenanigans. I’d put the Obama odds at 60-65%, not sure I’m as optimistic as some of the others.
Good insight, Drew. Thanks for the picture! This ties in nicely with Krugman’s blogpost “Asymmetrical Uncertainty” http://krugman.blogs.nytimes.com/2012/10/31/asymmetrical-uncertainty/
That was a neat plot. It definitely makes clear that somebody’s going to be really wrong. I do wonder whether the small firms are truly an independent metric to come down on one side or the other. Do they come up with their own methodologies? Or do they adopt what the larger firms use which would put them on the side of whatever house they come from?
We are relying on the scientific capabilities of the pollsters to do a good job. This keeps the hackers who steal elections at bay somewhat. But when the pollsters and officials fall down on the job, we have to rely on ourselves to make sure this election is not hacked and stolen. This has been done in the past. See the following paper done by professional statisticians and fraud experts who show that vote flipping occurred in the 2012 Republican primaries (benefiting Romney) and the 2008 Presidential election (benefiting McCain).
http://www.themoneyparty.org/main/wp-content/uploads/2012/10/2008_2012_ElectionsResultsAnomaliesAndAnalysis_V1.51.pdf
This could happen in this election!!
@Steven A
But was Rasmussen such an outlier in 2004? If I recall correctly, while others were wrong, Rasmussen mostly agreed with the aggregates. I could be wrong though. It’s also possible that Ras’ methodology for polling in 2004 was more accurate than it has been recently, perhaps due to the reliance on Robocalls?
David,
During election day, you can have a great time watching real-time as the intrade market responds to partial results.
I don’t want to get into an argument about market efficiency, but if you indeed believe the Intrade odds are off (for whatever reason), why not exploit that fact (we call it superior information) to make a killing on Nov. 6? I would. Ireland is a great place for a vacation paid for by statistical inference :)
Very interesting! I’ve been reading heaps of these pieces for weeks now. I am hoping that you are correct, but here’s my question. If I understand it correctly, the primary argument the Republican poll critics are making is that most polls are making the same assumption that the electorate will have greater minority representation this year than it did in 2008. The assumption would be based on the type of fact which would form the basis of almost any polling model — that is, that the demographics of the nation today (and for that matter, of registered voters) are closer to a “majority minority” than they were four years ago. However (goes the argument), 2008 was a Democratic “surge” year, and elements of the Obama coalition (including both young people and minorities) voted in record numbers relative to their numbers in the general populace. Most polling data indicates that there is less enthusiasm for the President today among those groups than there was four years ago. Ergo (the critique concludes), model that assume a higher percentage minority turnout this year than four years ago based simply on demographic changes would systemically overstate the numbers for the President. Can you answer this critique, or is the answer simply, “we’ll know on Tuesday night?”
Steven A,
an important structural change:
https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRAD4uiAhoEsrNfFK2dF8dMJUzVcR5npmAuuqcfvaepPk_L-cHnvg
So it appears reality really DOES have a liberal bias — at least this year.
dpendletonk:
Wow, I didn’t realize that the cell-only number was so *low* in 2004. I knew it was growing, but at only 5% during the 2004 elections.
Steve A: The answer is very simple. Rasmussen weights the polls to a likely voter model based on their own estimate of partisan percentages. i.e. if they survey and find 60% of their respondents say they are D and 40% R, but they think that the voter turnout breakdown will be 50/50, then they’ll “unskew” their result. They got 2008 wrong because way more (8%) democrats voted (many of whom where independents or republicans in 2004). This year they continue using 2004 as a baseline for their turnout. They will be wrong again…
In the polling this season it has also been obvious that right leaning pollsters have been releasing polls timed to push trend-lines more than the underlying trends. It would be interesting to look back at this election’s polling, and weight pollsters based on accuracy, then recalculate the pollster averages with those weights, and then look at trends in house effects. I suspect one would find that the “house effects” of right leaning pollsters is not constant but swings with the GOPs narrative.
More interestingly, I suspect one might find that it leads the narrative by a few days… The right leaning pollsters certainly seemed to run a pump and dump scheme on Obama between his convention and the first debate. Almost like they knew Romney would make the very unusual move of swinging back to the center in the first debate rather than at his convention (which clearly caught Obama off guard).
@ Peter Principle
Reality, to the extent that it likes to follow physical laws rather than biblical pronouncements, always has a liberal bias, not just this year.
In contrast, the Governors of Texas and Georgia, who believe prayer can overcome the physical laws that govern the atmosphere, have a conservative bias.
http://www.reuters.com/article/2011/04/21/us-texas-wildfires-idUSTRE73K7WY20110421
Hmmm… I’m suspicious of how those Gravis and Rasmussen polls align so closely. Given that Gravis is so new, could they be a front to distribute Rasmussen performed polls under a different name?
“the Governors of Texas and Georgia . . . believe prayer can overcome the physical laws that govern the atmosphere”
At this point, they might want to throw in a few prayers to the polling Gods — although in Rick Perry’s case, maybe not.
Kudos!
Steve A:
If Drew, 538, Sam Wang et al., turn out to have it right on Tuesday, I’d wager this will be the last presidential election for land-line only polling.
I would tend to take YouGov as the neutral point because, as a UK company, they do not have a dog in the fight.
I believe that all polls will be off because none are considering 3rd party. Johnson, in some polls preferentially takes about 3% from Romney and 1% from Obama. In others, he takes about equal from each. In still others, he takes his full strength from the undecided.
I have seen a report of a poll where he is winning WV. If so, what other states is he winning? How many Ron Paul supporters are going to vote for him? I know that in some red states a number of liberals are voting for Jill Stein.
The last poll with him in it from VA took 3% from Romney and 1% from Obama. It had Johnson at 5%, Goode 1% and Stein 0%.
My thought is that Rasmussen was “right” in 2004 not because they were smarter than others but because of latent factors, pollsters were systematically under-polling support for Bush. Since their house effect was to inflate GOP support relative to the norm, they looked good. In 2008, they were “wrong” because it was a good election for Obama, and the pro-GOP house effect led them to under-estimate Obama’s landslide. The strong house effect arguably limits their value as a pollster, since they’re not going to produce consistently accurate results. However, like a stopped clock, they may be right from time to time. Dr. Linzer’s analysis suggests they might be wrong this time. I hope he’s right, but the proof of the pudding will be in the election, which we’ll have real soon.
@LearningCurve
Running the risk of sounding like a broken record, the outcome of the election will not tell you what you hope. If Romney wins, that won’t mean Linzer and 538 were underestimating his chances. If Obama wins, what won’t mean that his chances were properly assessed, either.
The only way to know how accurate the predictions are is to repeat this exercise many times and then average the results. So, if you care to wait for 400 years, you will have a sample of 100 instances to know whether the projections were correct.
Therefore, no matter what the outcome on Tuesday, we will never know who is right.
It’s not bias, it’s a difference in turnout assumption/likely voter model.
Ras is historically very tight in his likely voter model. Tightness = skewed towards more republican-centered likely voter base. Ras will do comparatively well in years where turnout is down and we see a lower ratio of total/registered voters. The opposite in years where turnout is high.
This would explain a 2008 miss and 2004 hit. It turnout is low this year, that implies a republican heavy voter base and ras will do well.
You can call this bias; others would term it “proprietary likely voter model”.
This graph explains it well: http://www.bobkrumm.com/blog/wp-content/uploads/2012/10/turnout_2.jpg
I like the conclusion, but I’m not sure I buy the argument.
Your graph at the bottom does indeed suggest two distinct polling methodologies, one R-leaning and the other D-leaning. With you there. The truth may be one, or the other, or the average. Still with you.
But why should the aggregate results of the many small pollsters tell us which of the two methods is *better*? Aren’t you just measuring which is *more popular*?
@Allan Marlow
Nice bit of obscurantism you have going there.
The poll aggregators have looked at hundreds of polls, identified possible mechanisms (in the case of Linzer and Wang) for why they might diverge, and predicted the outcome with a reasonable confidence level. It seems to me that the results next Tuesday will pretty much make or break the validity of their methodology.
@ Partha Neogy
Sorry that the nature of statistical modeling sounds obscure to you, but unless you have the right training, that is very likely to happen – no offense meant. I am not going to enjoy a Romney victory, but I’ll be able to make sense of it.
@ Allan Marlow
For your eyes only. I do have a Ph.D. in physics from the University of Pennsylvania. And I do resent your patronizing tone.
@Allan Marlow
That’s not really correct. If the predicted probabilities were 60%, or even 70%, you’d have a point. But they are 83% (538) 97+% (PEC), 95+% (votamatic). With probabilities that high, a failure will tell us that there is very likely to be a problem with these models.
This election is going to be a defining moment for polling and poll aggregation models. Barring some crazy polling change over the next few days, all these models are going to predict an Obama victory with a high degree of confidence. If Romney nevertheless wins, the credibility of aggregating polls to predict election outcomes will be destroyed.
Well, perhaps it would really be the credibility of the large majority of pollsters that would be destroyed. We will have a clear demonstration of widespread systemic bias. Until the source of that bias is satisfactorily explained and corrected by pollsters, no one will trust poll aggregation. And they would be *right* not to trust it!
So basically, Occam’s Razor tells us that with the aggregate of the smaller polls agreeing with the left-leaning polls we have our answer:
The right leaning polls have a systematic bias and are wrong.
In fact, based on Rasmussen magically moving from +2 Romney to even, it appears they actually >knoweveryone else< in polling will have to do some answering for their failures. But if the election goes as predicted, it's going to be correct to ignore Rasmussen and the right-leaning polls in the future.
Welchi, you are wrong, they would *not* be right to mistrust it. Suppose your favorite basketball player has a free throw percentage of 98%. If he misses one shot, are you going to mistrust that he can be spot on most of the time? Of corse not! Think about it …
That being said, i agree that if Romney wins that the GOP are going to defame any models and modelers who called it against Romney, just like they’ve tried to marginalize science in general (and been pretty successful at it). But objectivity will wins in the end.
@ Partha Neogy
My sincere apology.
@ Weichi
In the social sciences the 2-sigma rule (about one in 20) is usually applied to observed data to sort out chance outcomes, in this regard you are correct .
When we talk about the validity of a model, however, the 2-sigma rule is not stringent enough, not by a long shot. Think about David’s argument for a minute and you see he is right. In the Higgs experiment, a 5-sigma rule was used, or a 99.9999% certainty level (or less than 1 in 3 million).
Forecast models don’t predict outcomes, they compute probabilities of outcomes. To test what they compute, therefore, you must be able to infer that probability by multiple applications of the same model. This makes it practically impossible to test the model, UNLESS the model uses a VERY HIGH sigma rule!
If Romney wins, conservatives will attack models and modelers, and just by sheer chance and totally unbeknownst to them, their attacks will be mathematically justified.
I think this is all moot.
Rove and Morris have started to walk their statements back.
Allan Marlow
I think 5 sigma was valid for Higgs, but is it really accurate to model carbon-based systems with DSP? What if the underlying structure of human reality is mandelbrot and not gauss?
(that is what Taleb hypothesizes)
@david
> Suppose your favorite basketball player has a free throw
> percentage of 98%. If he misses one shot, are you going
> to mistrust that he can be spot on most of the time? Of
> corse not! Think about it …
But this is a totally different thing! The free throw percentage is known because we have observed *many* free throws. But we’ve observed these election models only a very small number of times. That makes a big difference.
Continuing with the baketball analogy, let’s say that we have a player who we have never observed shooting a free throw. So we come up with a model predicting his free-throw percentage based on other aspects of his game and other measurements. Our model predicts that he will make 97% of his free throws. Then he starts shooting. He nails the first two shots. The third one misses. It is true that this doesn’t truly falsify the model, but we should now be very skeptical that the model is correct.
@ Wheeler’s Cat
The use of sigmas to convey brackets of event certainty doesn’t imply you believe in a Gaussian density for the event in question. It is used as a matter of convenience, that is all. If, as Taleb correctly assumes, reality is such that it contains much more “probability mass” under its tails than you can capture with typical continuous densities, all that means is the sigmas are no longer attached to frequencies from the normal density, but you can still use them.
As for AGW, that is not an experiment you have the luxury of repeating over and over to see if the models are right. In that case, you have to accept the fact that properly informed people are doing the right thing and trust them, there is no other alternative, just like you trust cancer researchers, aircraft designers, etc.
If you are a conservative, you will doubt that AGW happens, regardless of who espouses it, but if you are a conservative with a heart condition that requires immediate help, you won’t doubt the competence of your surgeon.
This dichotomy is not acceptable at a social level in an advanced nation. The fact that it happens in the US is what should scare all Americans who consider themselves civilized. Their time might just be running out, as it did run out of the Islamic intellectuals five centuries ago.
Oh, sure, sigma is still valid….what about precision though? My basic question which you tangentially address through AGW denialism is about the assumption of the normal distribution. Using Gauss at all is an approximation, right? Using DSP models to approximate carbon-based systems.
Like Dr. Linzer says, if there is increased systematic error in the lean-red group of pollsters and its NOT removed by averaging….then maybe the underlying structure of the data is asymmetrical. Even nonparametric statistics are presumed to go Gauss when the sample size is large enough.
“as it did run out on the Islamic intellectuals five centuries ago.”
I’d argue that if it wasnt off topic.
;)
You’d be surprised how congruent Ibn Arabi’s Time and Cosmology is with Sean Carroll’s From Eternity to Here.
By islamic intellectuals do you mean ibn Rushd and the faylasef? That is westernistic not islamic.
I guess im questioning whether carbon-based ‘reality’ is actually symmetric.
AGW denialism is a good example of asymmetrical ideology.
Wheeler’s cat
Let’s keep it on topic and simple – no need to worry about esoteric stuff.
Please download Linzer’s paper and read it. I can then point you to precisely the place where information is structurally missing from the model. It is very simple really, and the author is aware of the issue (as are the other modelers). Their problem is that by not incorporating all sources of information, the rules of statistical inference are tying up their hands, but this doesn’t have to be the case. Romney has much higher chances of winning, and it is NOT because polls are skewed or biased.
What is your background? (just to know how to communicate, that is all)
@ Allan Marlow. Please stop saying, “Romney has much higher chances of winning…” You are harshing my mellow, dude.
Dr. Wang is back.
/happydance
If I want to do a poll, where would I go to get a likely-voter model? Is there some “standard model” that everyone uses if they don’t have the wherewithal to develop their own?
If so, this could explain why most firms have the same apparent bias. It of course would not necessarily mean that this likely-voter model is the correct one.
I just visited PEC for the first time, what an eye opener it was! I never imagined a resonant chamber could be so, how can I put it, well, resonant. The complete inability to see or hear the outside world is something one expects to find among Limbaugh folks, but I confess that finding such an extreme example of self-deception at Princeton was a shocking experience.
What do you call someone who takes a walk, sees a 100-dollar bill lying on the sidewalk, nobody is looking, and keeps on walking without picking it up? That, my friends, is the very definition of stupidity. And that, my friends, is what all and every believer in the 98% Obama outcome probability is effectively doing.
Please, if you really believe the 98% Obama victory, why on Earth don’t you take advantage of that superior knowledge, open an account on Intrade and make a bundle of money, right now? You don’t because you know that 98% is nothing but utter and silly nonsense. Bad models lead to bad predictions. That simple.
The ability of smart individuals to deceive themselves is astonishing.
@ Allan Marlow
“if you really believe the 98% Obama victory, why on Earth don’t you take advantage of that superior knowledge, open an account on Intrade and make a bundle of money, right now? You don’t because you know that 98% is nothing but utter and silly nonsense. Bad models lead to bad predictions. That simple.”
If you think 98% is high your next stop should be Election Analytics at University of Illinois. According to them the probability of Obama’s victory is 99.6%
http://electionanalytics.cs.illinois.edu/election12/index.html
allan marlow,
I may have missed it, but I’ve seen you reference repeatedly your reason why these statistical models are “wrong” but not actually post it. What sources of information do you suggest incorporating? If polling aggregates are not good predictors of results, then I’d like to have some substantive argument why rather than blovation.
Unless the polls are systematically biased, which is possible, then Obama probably does have something close to a 98% chance of winning. PEC does not attempt to quantify the likelihood of systematic bias. Other more Bayesian sites, such as 538, have attempted to factor in a chance of systematic bias. If you believe that the polls are systematically biased, I’d be curious to know your basis.
I can only infer that you believe the markets such as Intrade are better predictors. These sites, which operate more like sports books than the financial markets, do contain valuable information, but there is little indication that they can be considered efficient. For example, Intrade uses Algorithms to automatically buy and sell contracts and book profits. Further, there are often persistent arbitrage opportunities that do not disappear. Right now, you can bet Romney at Intrade at 3-1 and at Pinnacle at 4-1.
Steve A
According to Rasmussen’s website, “After the surveys are completed, the raw data is processed through a weighting program to insure that the sample reflects the overall population in terms of age, race, gender, political party, and other factors.” Most pollsters do not re-weight for party ID, instead considering that to be one of the items that they are measuring in their poll. Party ID is volatile, so I’m not sure how you can do this and not show such observed inconsistency.
Commentor,
Let’s separate two issues: a) The correctness of the models, and b) the belief that the model numbers are right. I want to focus on b)
My contention is that modelers, as well as those who like the numbers the models produce, are not really intellectually honest when they claim to believe in the projections’ accuracy. There is not way to escape from this conclusion.
If you really believe the 98% Obama prob. is correct, then you should be expected to be enriching yourself in Intrade and similar places, AS WE SPEAK. It is as simple as that. If you are not, then we must ask, are you leaving piles of cash on the table because you are stupid, or are you leaving the cash behind because in your heart you know, or suspect, your 98% is nonsense. It is the latter. People who do research at Princeton, Stanford and similar places are not stupid, but they and and often do succumb to self-deception.
If you come and tell me you know a place where there is a room with gold in it, and that the door is open, and that I can go and take the gold, but you don’t do it yourself, then I can only conclude that you don’t really believe that story yourself. This is exactly what you are doing by telling me that Obama has a 98% prob of winning, and that there are investment markets where that prob can be exploited.
If I believed for a minute the PEC and similar numbers, I wouldn’t even be writing anything here, I would be investing, as much and as quietly as possible in the betting markets and would be increasing my wealth.
As to why the models are incomplete, that requires lots of hard work and can’t be discussed in this framework, obviously. As I mentioned before, we went through something similar with models in the credit markets when the crisis clearly showed our models were leaving out important information. I see a parallel of sorts here.
Allan Marlow is quite correct. The 98% chance (Wang) and 83% chance (Sliver) of an Obama victory simply don’t pass the smell test. Note the steady stream of commenters who say things like “I come here because your analysis comforts me.” That is a prima facie case that the visitors to these sites are anxious about the election, and intuitively sense that Obama’s odds are much lower. For an event like this, I think Intrade and Betfair useful. Good evidence suggests that people’s impressions of who will win are excellent predictors of election outcomes–possibly better than traditional polls. The odds markets are distillations of these sentiments. Intrade 2:1 for Obama, Betfair 2.5:1 for Obama.
Dr. Linzer. Thank you for your dedication to the statistical facts, though I freely admit many of the more complicated analyses are a little beyond me. It is also nice to come on this site and see some familiar names.
The fact that your site gives Obama the highest EV total is very comforting.
Thank you.
@AllanMarlow: I did go to open an account on intrade a month or so ago, but it was quite involved (requirement to fax multiple proofs of address to Ireland, etc.). I was too lazy to finish the process. Yes, I’m 99% sure that I’m leaving money on the table.
Also, don’t forget the concept of asymmetric utility. You’re so smart that I’m sure you know what that is.
Your criticism of the models appears to come down to unrecognized fat tails. Yawn.
And, my PhD is in physics from Stanford. Just sayin’.
MarkS
Perhaps opening an account on Intrade got more involved than it used to be, but you can hop on a flight to Ireland and stay at a hotel for a week. If you *really* wanted to exploit the opportunity you *believe* you have, you wouldn’t let anything get in your way. You managed to do your dissertation at Stanford, and you are too lazy to cash in 100 grand? Come on, tell me something more convincing – like you are not all that sure after all :)
I am sure Allan. Obama by 4.6% in the popular vote and 332 EV.
But I’m a Sufi and I don’t gamble.
/that’s what *she* said
;)
@AllanMarlow, the comparison to credit market models is invalid. The credit market models were based on the assumption that the underlying collateral would not decrease in value. When combined with the dangerous levels of leverage and the utter insanity of the synthetic collateralizations, the models were blind to the underlying risks. Also, a compelling argument can be made that the models were willfully blind, since the incentives were biased in favor of ignoring the underlying risks, as the money to be made was extraordinary.
Unless you can offer some concrete evidence that the underlying collateral in these models (the public opinion polls) are systematically biased in Democrats’ favor, then the models will probably be correct, having worked successfully in the last one or two presidential elections (depending on the model).
As to why we don’t cash in on the prediction markets, the arbitrage opportunities are very limited. It is estimated that it took $10,000 on InTrade to reduce Obama’s odds from 60% to 50%. How much do you estimate it would take to drive Obama’s odds on InTrade to 90%? The amount to be made on the betting markets is trivial.
Allan Marlow,
Your rant against PEC is almost a straw man argument seemingly designed to avoid any substantive argument. Sam Wang’s site readily notes:
“These calculations would be affected if there is an overall poll bias, which can have a large effect in a close race. Bias could happen if polling methods do not accurately sample actual voting patterns.”
I’m not sure that even Sam Wang would bet on his model being 98% correct, so it may be time to retire this argument.
Allan: asymmetric utility! Obama is currently at 6.62 on intrade, so to “cash in 100 grand” I would have to put up 200 grand. I would have to send it to some people in Ireland, and trust that they’re not going to misplace it, or collect it and disappear, or whatever. I just don’t have that much trust in whoever owns/runs intrade. (If it was the feds offering T-bills, that would be another story. Or Vanguard. Or Fidelity. Well, maybe not Fidelity!) And my negative utility for losing 200 grand is WAY bigger (in magnitude) than my positive utility for an extra 100 grand.
Ok, lets finish the argument with this note. If you project 98% and the underlying assumptions have a p% probability of being wrong (prob of polls biased, etc.), then you should report 98% * p as your final result, with the caveat that p could be large. That would be a different story. What I don’t like is the degree of intellectual dishonesty in all this. By the way, 98% * p = 64%
Allan: your accusation of “intellectual dishonesty” is ludicrous. Everyone acknowledges the possibility of unquantifiable “systematic error” (e.g., the polls are just wrong, Republican voter suppression works, etc) in the models; see, e.g., Drew’s title for the post we’re all commenting on. My personal probability that Obama wins is around 95%. I don’t bet though, because of asymmetric utility and my lack of trust in the people running intrade.
My mellow has totally been harshed. WTF, why don’t we all just wait until Tuesday evening? We’ll all know the answer in less than 72 hours. Then we can all send smug emails to everyone who disagreed with us. ‘K?
Anyone else starting to get the feeling that Allan Marlow is not so much upset about intellectual dishonesty as he is the results not being favorable to his desired outcome?
I may be wrong …
His non-statistical argument is that of a 12 year old. “If you’re so sure, then I’ll bet you $20 that you can’t do ____.”
I have maybe $2000 extra bucks in my bank. Times are hard. Yeah, maybe I could double it, but I may need that money tomorrow. I’m not going to gamble that. In fact, I don’t believe in gambling anyway. I suspect I’m not alone in this.
I disagree that the binary Presidential EV outcome is the only tool we have to assess the accuracy of the pollsters and poll aggregators. Can we not also look at how well they predicted the 51 state outcomes? By “predicted” I mean: estimated actual vote distribution? In fact, can’t we even compare the sigma of the various individual polls and aggregates to the actual errors, to see if how well they estimated their precision?
I think we have heard enough from Allan Marlow about this Intrade crap!
What Allan does not understand is that there is much more than money at stake in this election. Our future is being decided in places like Ohio and Florida and the rest of us want some idea of whether we are going to have to apply for Mormon citizenship under the Boss Man Romney.
So Allan. This is the word. You are done friend. No more posting on this site. No more posting at PEC and leave Nate Silver alone.
If you ever had a point you have made it. These sites are not the typical troll sites. Go to MSNBC if you want to fight with “Feisty” and the other liberals who love a good brawl. This is a civilized site (my remark about Mormon citizenship is the exception)!!
@Allan
If you are so convinced that Romney has better chances than Obama, surely you should be betting on Romney on Intrade or any of the other websites? After all, the odds on these websites are all in O’s favor. If Romney is the true favorite, as you say, then you could just make an absolute killing betting on Romney, yes?
@Allan
Even if I am 98% sure that Obama will win, I am not 100% sure that a bet that I make on Intrade will prove profitable. That’s because I am not even 50% certain that there is no “counterparty risk” – http://en.wikipedia.org/wiki/Counterparty_risk#Counterparty_risk – when dealing with Intrade.
So, let’s say that I bet 200K on the election result and I made 100K profit, would you guarantee that YOU would pay me the 100K in case Intrade failed to pay me? What is the collateral? Perhaps your house?!
@David,
You could not double your money, you might be able to make 720 on your 2000 dollar bet if you could space our purchases out enough not to drive the price up.
@Trim,
I think what Allan Marlow is doing is saying that he too believes Obama is more likely to win than Romney, but with a lower probability that projected by others. In other words, he’s come up with an argument in which he can claim that have been right no matter the outcome.
dpendletonk, Steve A, Hedgehog: It’s entirely possible that the smaller firms are making the same methodological “mistakes” (whatever those may be) as the comparable larger firms. But what are the chances? As you say, we’ll know soon.
I’ll just add that most pollsters don’t model the electorate, exactly. What they do is take a random sample, and then weight to correct for demographic discrepancies between the sample and the known population – factors such as age, race, sex, etc. Sometimes they’ll vary in how they screen for, or define a “likely voter” and this is another source of house effects. But mostly this is done by asking questions like “how likely are you to vote?” which is fairly innocuous. It’s when they start making assumptions about what the party ID or demographic composition of the electorate is going to look like that they can really run into trouble.
Allan: I’ve been following your comments, and I appreciate them. I do think that on Election Day, we’ll be able to evaluate and compare the models, and then use that to improve them (to the extent it’s necessary). The main thing to be looking at isn’t the overall EV forecast, but the 50 separate state outcomes. I’ve already laid out the standards I’ll be using to check my results.
You’re probably familiar with the old George Box saying, “all models are wrong, but some are useful.” That goes for what I’m doing here. I’ve never hid that my model assumes no net house effects on Election Day, and error in individual polls due only to random sampling variation. The question isn’t whether these assumptions are true – I know they’re not – but whether reality is so different from the simplifications in my model that it renders my forecasts uninformative. As you know from reading the paper, when I tested my model on data from 2008, the consequence of these assumptions was about 10% overconfidence in the state vote forecasts; otherwise, it worked great. The average house effect was very close to zero, and the amount of overdispersion caused by house effects was relatively minor. But that’s only one election. It’s why I’m very eager to see how well the model performs this year.
So I do agree that my probabilities aren’t meant to be taken as “absolute” truth. They’re model dependent, conditional on the assumptions in the model. I could add noise to the forecasts, but how much, and on what theoretical basis? It’s not clear. There was a nice piece by Simon Jackman recently that tried to address this question in a systematic way, and may be worth reading.
Hi Drew, I for one am glad that you are publishing the probabilities output by your model without adding in extra fudge factors for model error – I understand part of improving these models is being willing to go out on a limb rather than hedging your bets.
One of the immediate problems I have with any comments by Allan Marlow is his patronizing assumption that EVERYONE is driven by a profit motive.
Personally, I have no interest whatsoever in betting on this, or any other, election.
Using a CAPM model with rough estimates (I don’t have time at the moment for a more thorough analysis), it appears that markets will underestimate most models, since the polls experienced swings (both ways) past the margin of error over the past 6 weeks. In that sense, intrude prices seem reasonable.
@ Drew
As you correctly point out, these projections don’t rely on directly modeling the electorate. This is what in other fields is referred to as a reduced approach. An alternative approach is to characterize the “response function” of the electorate, either at the group or individual level (this done to some extent in sophisticated marketing.)
The latter approach is probably impractical throughout the election cycle, but some version of that must be present as election day approaches to capture any new versions of the Bradley effect (such as some social segments getting more invigorated than others at the last minute, etc.)
This characterization is what crowds can do, because they are the electorate and are not inhibited by social mores. While the models tighten on a daily basis, the market spreads are widening. This is precisely the opposite of what you would expect of models whose accuracy is supposed to rise, rather than fall, as the event approaches.
Agree that at the state level you will have an ensemble of data points to work with, but it is that pesky little piece at the end of the diffusion that holds the baby in the bathwater…
You can also do error densities by polling medium that is 3 distributions for live interview telephone, machine telephone (robopoll), and internet.
I note that Gravis and Rasmussen are robopollers. My sense is that most of the small pollsters use live interviewers calling telephones (the cost is part of why they publish few polls). It is well known that robopollers (other than ppp) have Republican house effects.
My sense is that internet based polls have Democratic house effects. Frankly this seems almost inevitable to me (no matter how you weight for demographics you can’t get rid of the correlation between dislike of newfangled things like computers and presidents whose middle name is Hussein).
I’d like to see the 3 kernel densities.
BetFair has Obama at about 79%, Intrade at 66%. Why? If the crowd is so smart, there shouldn’t be this wide a discrepancy.
It will be interesting to see Tuesday: did the polls fail or did the markets fail? I’m betting that it’s the markets.
(arbitrage should prevent such discrepancies from persisting, but it’s been this way for days).
@Alllan: what are the demographics for intrade etc.? I’m guessing that women and minorities are going to be under-represented, and that brokers/investors will be over-represented,relative to actual voters. This would again tend to give overinflated values to Romney-wins positions. Only 2-3 days to find out.
“Because you’re proceeding on your feelings. This is Intrade; if there’s a rogue buyer it’s a tremendous opportunity to make money. There’s a real high duhh factor to that, so I don’t understand why there’s so much kvetching when these Romney whales operate. It does not make any sense.”
This came directly from Intrade traders. When whales have the ability to manipulate the numbers with out a higher cost for doing it. Also this is a nice way to distribute your campaign donations across new mediums. In this case we may actually be seeing a coordinated market failure which is kind of funny. That is the definition of a market failure and is obviously the biggest problem with these prediction markets not enough volume to do anything meaningful but there people out there making money on this it is just tough to make anything meaningful.
David
Betfair is much bigger and very British. I think the Brits remember #romneyshambles and #AmericanBorat.
could be bias.
;)
Maybe you can answer my question, David– why would anyone believe that the Market would inform the Math? Markets are subject to Irrational Exuberance and trader manipulation.
Unlike the Math.
As a follow up on Intrade what is occurring reminds me of something I used to encounter as a Stock broker. Because of the illiquid nature of Intrade the community seems to have identified something resembling a Pump and Dump scam making the actual numbers unreliable from a predictive standpoint because the traders moving the price are not working under a profit motive as it pertains to the contracts. This is an issue that people have hinted at in the past but this may be the first time we have seen it in action live. It makes rational sense for the whale if the money they are intentionally losing was money they were going to contribute to the campaign anyway this would explain the fact that the arbitrage opportunity has not been taken right now someone has set a ceiling on the contract at 6.66 with 8000 contracts at that price. It is actually brilliant if the money lost is not meaningful to the people involved and the fact that Intrade gets covered more than the other markets would explain why it is the outlier for Romney.
Drew, I’m having a LOT of trouble following Allan’s….umm…argument.
He *seems* to be saying that Romney’s win prob is actually ….umm…Intrade odds?
Is that correct?
Is he trying to use market utility and bidding theory to prove the state polls have systematic error and overestimate Obama’s win prob?
Wheelers cat
The market is the benchmark with which we calibrate the math – at least four people won the Nobel in Economics for this realization.
David
I don’t know the demographics, but arbitrage persists because trading across accounts is impractical.
“The market is the benchmark with which we calibrate the math”
say wut? how can you ‘calibrate the math’ with a benchmark that is subject to manipulation and irrational exuberance and partisanship and cultural trends?
Isn’t it somewhat overparameterized?
Of course, im just a sillie grrl, and probably not capable of understanding cartels and regulatory capture and utility theory and tail-risk hedge funds and such.
;)
Well we can stop this whole discussion of markets versus statistics.
The Redskins lost today. Everyone knows this means that the President will lose. Of course, the NFC won the SuperBowl and the means the President will win.
So it all comes down to this — who do you believe??? The New York Giants whose win guaranteed Obama 4 more years or the Redskins who seem to do the same for Romney??
Allan Marlow,
“The market is the benchmark with which we calibrate the math – at least four people won the Nobel in Economics for this realization.” The literal translation of your statement is that any mathematical model that does not comport with the market is by definition inaccurate. This would make sense in the financial markets, but seems like a massive logical leap in this context. Question: which market should the statistical methods calibrate to? Intrade? Betfair? Another? And if the polls stubbornly disagree with the market, should we simply disregard them? I find this to be a profoundly absurd statement in this context.
If trading across accounts is impractical, then isn’t that an admission that there are substantial transaction costs and therefore inefficiencies in the markets? Isn’t that statement completely antithetical to your first?
Commentor
I wasn’t saying electoral projection models should be calibrated to markets, I was simply answering Ms Wheeler’s Cat’s question. Beyond that, yes, you should incorporate information from markets in your model, if you find a proper way to do it (the proper way to do this must be studied.)
I agree there are issues with Intrade in that it takes time to enter and unwind, but the transaction cost is not the primary issue, the fact that increasing your collateral takes time is.
Today there were significant outstanding selling orders in intrade that kept the Obama price from changing much. You can’t say that was manipulation. It is hard to find volunteers to put up their cash to manipulate markets.
As an example, Limbaugh might say whatever he wants against Global Warming, but I bet he is not betting HIS money against it. That is what is nice about markets, they force the truth out in the open!
“I wasn’t saying electoral projection models should be calibrated to markets”
Oh yes you did. Drop the weasel wording and take your lumps. You are arguing for some nutty reason that is opaque to me that the markets give a more accurate picture than the state polling.
I just dont think that is true.
You asked about my background.
Its games theory.
@wheeler: “Maybe you can answer my question, David– why would anyone believe that the Market would inform the Math? ”
Well, it’s not a one way street. And I agree with FalsifyPopperism’s critique. Because there is no such thing in life as a free market or free lunch. Markets have rules and regulations structuring the market and they have participants and regulators (even Intrade). These determine the structure of the game (from the game theoretic view). How rational and/or collaborative/competitive are the players? Externalities (such as arbitrage in other markets; and whether some player(s) are lying) further complicates the picture.
But let’s say you use something like CAPM for portfolio management. That is, we have a mathematical model which (hopefully) contains all the relevant variables. Often there are parameters that are either unknown or grossly estimated (for example, in Newton’s theory there is the model v = -g*t, where we have the gravitational parameter g). By making many observations, we can estimate the value of g, thus calibrating the model. But then we may find the model fails us outside of the simple situation where there is no atmosphere or that air resistance can be ignored. Then we must correct or rebuild our model so that it fits to reality. The process is: build a model with parameters; the parameters are calibrated; the model is checked against reality; and repeat.
As far as I can tell, Allan is assuming there is an implicit model in the group psychology of the intrade crowd, and that by trading they have reached an optimal approximation of the value of the EVs relative to 270. So there is no mathematical model, it’s like some multi-agent genetic algorithm constantly seeking to optimize expected return.As you (wheeler’s cat) mention, there are plenty of reasons for doubting the results. The housing bubble crash of 2006-2007 leading to the grand recession is but one case where market players ignored publicly available data that even in 2002 and 2003 showed that the skyrocketing home prices across the nation couldn’t possibly be sustainable. You can’t ‘calibrate’ out actual data just because you wish for something that’s good for your pocketbook, in other words. Math/accounting wins out over markets when market speculation is not based on solid fundamentals and very careful risk analysis. Caveat emptor.
David
With all due respect, your last paragraph above is pretty much mixed-up nonsense, my friend. I don’t doubt your good intentions and intellectual honesty, but keep in mind that if you venture out too far speculating on stuff outside your area of expertise you run the risk of sounding silly :)
The thing is, Intrade is not an efficient market because the volume of trading is too small meaning one deep pocketed idiot can skew the odds. We will know in less than 48 hours.
@Allan Marlow. I dunno, David’s post made sense to me. You’re the one who’s holding a position that everyone else seems to disagree with and that you are offering no strong supporting evidence for. Saying you’re right and an expert and that the little people wouldn’t understand your argument doesn’t really cut it. Sounds a bit like Romney and his tax plan…..
Thankfully in a couple of days, state poll aggregation will have been proved to be sound and the betting markets less so.
ZSdust
If you can set aside a couple of years to get advanced training in finance, please do that and then report back, I’ll be happy to answer your questions if you still have any lingering doubts.
@Allan Marlow
There you go again.
Look, it’s not that complicated: What do we know about whose going to win an election? Not much, other than what polls tell us–which it turns out is quite a bit. So, either you can judge the general gestalt from what you see on TV–ie pundits talking heatedly about the polls–and then go bet some money on who’s going to win OR you can develop a statistical model that crunches all the polling data and maybe tosses in a few other factors.
I choose to go with the expert statistical analysis; you choose to go with the wisdom of the crowds, based–i think–on the fact that they have money riding on their choice. I, and most others here, don’t agree that this makes their prediction more precise.
I’m extremely impressed that you have a couple of years of financial training, just as you are probably impressed that I have two masters and a PhD. Congratulations on your achievement, and see you Wednesday!
David.
I’m super bored with Allan’s argument from authority.
From a game theoretic perspective the Market is subject to a host of manipulative influences like cheating, cartels, irrational exuberance, etc.
Its not a benchmark for anything.
@wheeler: remember all those physicists who reacted vehemently to Einstein’s new theory? But then the empirical measure of the perihelion of Mercury proved Einstein right? In a similar fashion, the finance community seems to be in denial that the markets aren’t efficient and are prone to instability (read Kuhn’s “The structure of scientific revolutions” if you haven’t, fascinating stuff). The empirical evidence is clear, yet they cling on, for to do otherwise would mean their investment of time and their professional standing would be devalued. The bloated financial sector is on the gravy train (they do well even in economic downturns). So of course they have a huge economic incentive to argue vehemently for what is provably wrong.
If you want a pretty thorough list of all the ways the markets violate any reasonable hypotheses for efficiency, stability or agent rationality, see for example, “Investor Psychology and Asset Pricing” by Hershleifer. (http://mpra.ub.uni-muenchen.de/5300/1/MPRA_paper_5300.pdf).
As far as the predictive power of the market, it’s also important to remember the structure of that market. The incentive in the Intrade market is (presumably) to maximize the individual position holder’s return. And it’s a one off: the trading ends for all time at a specific time with a specific pay-off, which means this market should behave like a bond market rather than a stock market. Except the probability of default for the bond issuer (the candidate, in essence) is fairly high compared even to state or municipal bonds. Thus a $10 bond for the candidates are trading at about 33% and 66% discounts, with the chance of default (according to the national polls) being about 50%. According to PEC, however, the risk of default on an Obama position is estimated to be 1%, 15% over at Nate Silver’s, etc.. So, depending on how risk averse you are and which metric(s) you believe best model the default risk, you can place a bet (if there is a counterposition available). Looking at the markets as they are, it’s pretty clear that the players are informed by the polls and not by some “invisible hand” floating through, with some personal bias sprinkled in for good measure. So you can expect these markets (due to investor conservative behavior) to tend towards the center more than the polls indicate, since the polls could be wrong (can’t wait to find out!).
Now, I’m sorry that my writing style might not make sense to Allan, who has expertise in a non-technical field, but I’ll be glad to continue to refine my explication.
Intrade market manipulation happens, but doesn’t have persistence:
http://www.washingtonpost.com/blogs/ezra-klein/wp/2012/10/23/how-to-manipulate-prediction-markets-and-boost-mitt-romneys-fortunes/
Intrade gets things very wrong sometimes:
http://www.ritholtz.com/blog/2012/06/healthcare-upheld-by-scotus-intrade-blows-it-again/
Given that Rasmussen appears to have a slight Republican bias, and PPP appears to have a slight Democrat bias, you can find presidential election snapshots using only each of their respective polls for the battle ground states. Visit http://electionanalytics.cs.illinois.edu/ for details.