On polls and election predictions
May 15, 2013

On polls and election predictions

If nothing else, the BC Liberals' entirely unexpected and dominating win in BC last night will be a useful lesson in humility for pollsters and those that rely on them to prognosticate about elections. All the polls showed a strong (and possibly growing) lead for the NDP, and poll aggregators like threehundredeight.com were predicting a 98.3% chance of winning the popular vote and the actual results were outside of the predicted likely seat range. Clearly the polls (and thus any model based off the polls) were horribly wrong. Yet on the flip side with the last US Presidential election we saw that the polls in general, and aggregators like the Princeton Election Consortium and 538 in specific, were incredibly accurate. What gives?

Polls have always been, at best, a proxy for measuring what we are actually interested in: how people are going to vote. We know there are numerous distinct biases that come up with polling (such as calling landlines gives a different set of people than calling cellphones), and pollsters and aggregators both attempt to adjust for those biases. To some extent, especially, when there is lots of good data, these can be adjusted for, but never completely and at times the biases can be surprisingly strong in a particular arena.

The value that an aggregator gets is often overstated. Nate Silver of 538 was viewed as a sort of mathemagician for his prominent display of a very accurate prediction (although not the most accurate out there). The more down to earth reality is that creating a poll aggregation model probably can hope to increase the accuracy of a prediction by a few percent. Indeed, in the US Presidential election case, simply taking a rolling average of polls in each of the states was also a pretty accurate predictor of the outcome (partly because the pollsters themselves do a lot of the adjusting for likely voters and the likes themselves). Having a ridiculous complicated model, as 538 and others do, that adjust a range of parameters due to various correlations, imput economic data, etc may help. But only a bit. Enough, perhaps, to make the right call in a pretty tight election with very good polling data, but nonetheless not a huge difference.

The same is true in BC. Eric Grenier at threehundredeight's model (although far simpler than what is done a PEC or 538) can hope to be, at best, a little better than just averaging polls. And it was. However, the reality is that polls themselves were so grossly wrong there is nothing he can do about it. For a US Presidential election there is an enormous deluge of polling information, with dozens of different polls released today broken up into different regions. Compared to the trickle in BC, it is an amazingly dataset and gave amazingly accurate results. Polling for a minor provincial election in Canada is just simply of far worse quality, and we have seen that manifested here today. Indeed, the simplicity of Eric's model is partly a necessary reality given that he doesn't have the extra data to do some of the more complicated stuff.

It should be noted that adjusting in a bad way can cause harm. The Romney camp, and much of the GOP establishment, was making predictions much worse than the average of the polls because they used a different likely voter model that gave different results. These errors cost them dearly.

The real value in polling:
We all have an insatiable desire to know the answer to unknowns (as a poker player, I distinctly profit off the tendency of poor players to call just because they desire to know what I have, even when it is clear they are likely beat). Who wins an election is a big deal, and we want to know the answer to who will win as soon as possible. People who prognosticate about elections are filling the role of satisfying that desire, which is why they get attention. Outside of that, however, the value in predicting election results is rather limited, and has limited impact on our political scene (there is the issue of predictions feedbacking into the dynamics of the race and how people vote; I will ignore this because the mechanisms for all of this are so unclear at this point in time).

That said, I still believe there is incredible value in the polls, but this is because as a political analyst I am interested not just in the end result of who wins and election, but also in knowing how people react to various political events. Polls are very good at that, they are very good at demonstrating that there is or is not a sharp change in direction in people's sentiment when something happens (a gaffe, a release of a platform, a news event, etc). Nate Silver has said similar things before, that much of the value in regularly reading a blog like 538 is not in that it is slightly more accurate in predicting the election the day before the election, but to give an indication of how well the messaging worked at the Democratic National Convention, say.

Marginal vs absolute value:
The fundamental dynamic going on is that polls from the same company done in the same way give a very good measure of the marginal changes. We can see that over a period of time there was a change of a particular magnitude in a particular direction. They are reasonably good at that. They are much less good, however, at having an accurate absolute value. As in, if we see an uptick in poll averages from 50% to 55% we can be quite confident that there has been a marginal change in support. But we are much less confident that the actual support rests at 55%. Whatever biases exist to make this poll a poor proxy remain in both measures, so while the 55% may be wrong because of the biases, that there has been this change occurred when the biases remain constant is significant.

Eric Grenier's Mea Culpa:
I was impressed to see that Eric gave an appropriately humbled questioning of the validity of his work in light of this result. It is refreshing to see such honesty in the face of objective results. However, I actually think he goes too far. Or at least, if one has already reduced one's expectations to that which I have discussed above, he has nothing to apologize for. His work certainly still has value, it just isn't necessarily of the value one might hope of being able to accurately call every minor Canadian election within a couple percent.  I will close out with the copy and paste:
"It puts into question the validity of the work I do. I write about polls every day for this site, for The Globe and Mail, for The Huffington Post Canada, and for The Hill Times. I give radio and television interviews about them. It is my full-time job. I've always approached it as a professional and have tried to provide insightful analysis of polling, separately from my role as a forecaster. No one in Canada who doesn't work for a polling firm writes about polls as much as I do. 
How can I credibly continue to do so when I myself doubt that the results are reliable? While I was shocked when I saw the results last night, a part of me was not surprised that I was shocked and that they got it wrong all over again. If I go into every election assuming that disaster is more likely than triumph, what is the point? 
This site was meant to be a way to cut through the confusion in polling and give a good idea of what, as a whole, the polls are saying. The site can still do that, but if what the polls are saying is not reflective of reality, what use is it? 
My projection was wrong because the polls were wrong. Again. I am sorry that it was so. I can blame the pollsters for providing me with unreliable information, but I am nevertheless responsible for what is posted here, for the defense of polling I have mounted for the last few years, and for whatever confidence I expressed when analyzing the numbers in an attempt to inform readers about the state of the race in British Columbia and elsewhere. I apologize for that. Where do we go from here?"

Thoughts on this post? Comment below!

Share this post:

Tweet It! Facebook Add Feed Reddit! Digg It! Stumble Delicious Follow

1 comment:

Anonymous said...

I voted NDP. But as I stood in the booth, with the pencil in my hand, I hesitated. I though, "Can I really vote NDP?"

That hesitation was born of Glen Clark's NDP's appalling incompetency, and their legacy of written off ferries and economies.

I wonder if many people polled as NDP, then at the booth, Glen Clark's spectre reached out and turned them back.

Post a Comment

Frequent Topics: