Tuesday, March 25, 2014

Explaining the difference between TNS-BMRB and the online pollsters

In the last post I discussed the ongoing disparity between TNS-BMRB, which currently shows a Yes vote of 40% after undecideds are excluded, and the three Yes-friendly online pollsters that are currently clustering around a pro-independence vote of 45-47%. I wondered to what extent the difference is caused by the fact that TNS are not reliant on volunteer online panels, and to what extent it's caused by them applying an eccentric weighting procedure. (They scale up people who didn't vote in 2011 or can't remember how they voted so that they count as a full 50% of the sample, which is a much, much higher figure than any other pollster uses.) So in order to test this important question, I thought it might be useful to compare the TNS cross-breaks with those of Survation and Panelbase. I'd like to have included ICM as well, but those datasets haven't appeared yet.

In order to make a meaningful comparison, it's necessary to strip out the Don't Knows, because TNS always report a far higher number of undecided respondents.

PROPORTION OF 2011 SNP VOTERS WHO SAY THEY WILL VOTE YES :

TNS-BMRB : 79.6%
Panelbase : 78.7%
Survation : 82.0%

PROPORTION OF 2011 LABOUR VOTERS WHO SAY THEY WILL VOTE YES :

TNS-BMRB : 17.6%
Panelbase : 32.3%
Survation : 27.9%

PROPORTION OF 2011 CONSERVATIVE VOTERS WHO SAY THEY WILL VOTE YES :

TNS-BMRB : 6.3%
Panelbase : 8.2%
Survation : 7.8%

PROPORTION OF 2011 LIBERAL DEMOCRAT VOTERS WHO SAY THEY WILL VOTE YES :

TNS-BMRB : 17.1%
Panelbase : 27.3%
Survation : 11.3%


So there's good news and bad news here. The bad news is that the bizarre TNS weighting procedure plainly isn't the only explanation for the disparity - they really are finding a significantly lower number of Labour voters 'on the ground' who say they will vote Yes, and that's clearly having an impact on the overall numbers. That would be the case even if they weighted their raw data in exactly the same way as Panelbase and Survation.

But what I find fascinating is that TNS are showing an identical position to the online pollsters in respect of people who voted SNP in 2011. This goes completely against the conventional wisdom about why online polls might conceivably be skewed towards Yes - ie. that highly opinionated, die-hard nationalists are signing up to online polling panels in disproportionate numbers. If that was the explanation for the higher Yes vote, you'd expect the difference to show up in the SNP column, not the Labour column. Why would the supposedly rogue element in the online polls - the surplus numbers of opinionated nationalists - have voted Labour in 2011? It doesn't make any sense, so clearly there is something more subtle and interesting going on.

The solution to the polling puzzle in this campaign therefore seems to hinge to a large extent on understanding the greater tendency of online pollsters to pick up a substantial Yes vote among Labour (and possibly Liberal Democrat) voters. Could it be that such people feel a bit more embarrassed about admitting their support for independence, due to their political backgrounds? Do they find it easier to be honest to a computer screen than to a real live interviewer? That's just one possible explanation out of many (another is that Labour voters who use the internet more often are likely to be disproportionately well-informed about the issues), but if it's the right one it would be fantastic news for the Yes campaign, because it would suggest that the online polls may be considerably more accurate.

3 comments:

  1. Could some of it be down to the question, TNS asking how people would vote in September rather than tomorrow? If Yes-leaning Labour voters are more likely to say "don't know" then they'll have been omitted from your figures about. What doesn't make sense is why that same group would say "Yes" rather than "No" if asked how they'd vote tomorrow.

    ReplyDelete
  2. Well, ICM and Panelbase both ask how people will vote on the actual referendum date as well. Of the three Yes-friendly online pollsters, Survation are the only one who ask how people would vote in a hypothetical referendum now.

    You're right that TNS are finding that Labour voters are more likely to say they're undecided (27% compared to 23% of SNP voters, 12% of Tory voters and 12% of Lib Dem voters).

    ReplyDelete
  3. Good analysis work you've been doing, James.

    The point about people who use the internet a lot being better-informed is a very good one, and food for thought for yes. The biggest risk for us arguably comes from two groups who aren't big internet users - the elderly and the working poor.

    When canvassing we've found that areas with high unemployment / casual employment rates lean more to yes, and people have more time on their hands to mess about on the internet. A broadband connection is cheaper entertainment than Sky telly.

    We have a much tougher time in areas where people are working long hours to barely make ends meet. Those people feel they have more to lose, have often bought their council homes, and don't have the spare time to read hunners of blogs.

    The extent to which we take the campaign off the internet and onto the streets during the summer will be a huge factor in the result, in my opinion.

    ReplyDelete