What the polls do and don't say

“Poll Points to Tory Victory” says the headline in the The Vancouver Sun, referring to the latest Ipsos Reid figures. The Sun reports that the Conservatives now have the support of 35% of eligible voters, compared to 31% who back the Liberals. “How Harper Fashioned His Lead” is the The Globe and Mail’s headline. The paper tells us that, according to Strategic Counsel, the Conservatives have a two-point lead over the Liberals, 33% to 31%. But are the Tories really ahead, and do the polls even say they are ahead? We have to look past the headlines and read the fine print.

The fine print is all about the margin of error. Although news organizations invariably tell us the margin of error for polls, they do so only as an afterthought, and only after invariably misleading us with the headlines. Neither Ipsos Reid nor Strategic Counsel actual claims that the Tories are ahead of the Grits. Based on a sample of 2,004 people interviewed, Ipsos Reid gives a margin of error of 2.2 percentage points, 19 times out of 20. Strategic Counsel, based on a sample of 1,500, gives a margin of error of 2.5 percent points, 19 times out of 20. What Strategic Counsel is saying is, “There’s a 95% probability that the Conservatives have the support of somewhere between 30.5% and 35.5% of voters and that the Liberals have somewhere between 28.5% and 33.5% of voters.” The Conservatives could be as low as 30.5% and the Liberals could be as high as 33.5%. That’s a large overlap. And even that is assuming that this isn’t one of the one-in-twenty polls that, though done correctly from a methodological standpoint, gives a wonky result outside the margin of error. Even the Ipsos Reid poll, which has a smaller margin of error, and which the Sun would have us believe puts the Tories up by 4 percent, does nothing of the kind. That poll, again with a 95% probability, puts the lower limit for Conservative support at 32.8% and the upper limit for Liberal support at 33.2% - a small overlap, but an overlap nonetheless.

It’s a mistake to think that if one party’s reported lead over another one is greater than the margin of error, then it must really be in the lead. Even leaving aside the 1-in-20 possibility for an outside-the-margin result in a properly conducted poll, it must be remembered that the margin of error applies to the figures for both parties. So if there’s a 2.5 percentage-point margin of error, and one party is reported as being three or four percentage points up on the other, there’s still an overlap. It's not clear to me that a piece called “Understanding Polling and Margin of Error”, at the Canadian Opinion Research Archive, isn't guilty of “innumeracy” in this regard. None of this means that the Conservatives aren't probably ahead of the Liberals now, but it's far from sure.

For new poll results and some interesting seat projections, check out the Hill and Knowlton Election Predictor. Remember, though, that (1) party support is spread very unevenly across the country, a fact that has huge implications for who wins how many seats, (2) smaller sample sizes for regions or provinces mean that poll results at these levels have to be taken with a very large grain of salt, given the large margins of error involved, and (3) large differences in voter makeup and attitude from riding to riding, combined with three- or four-way voting splits, mean that knowing how the parties actually do stand in a province wouldn’t tell us much about how a particular constituency is likely to go. The Election Prediction Project looks at things riding by riding, though in a pretty unscientific way.