How much credibility do the polls have?

Over the last couple of days, I've been contacted by three different people asking for my thoughts on this blogpost from Auld Acquaintance, which offers a number of possible reasons for believing that the polls may be understating the true strength of the Yes vote. So I thought it would probably be simpler just to give my reaction in a new post.

First of all, though, a few general observations about where we currently stand. As regular readers know, I get a bit exasperated with the 'Flat Earther' tendency among some Yes supporters in their attitudes to the polls. How often do we hear "well I never look at the polls, they're all rubbish anyway" or "the only poll that matters is on the 18th of September" or "OK, they asked 1000 people but what about the other 4 million?" It really doesn't do any of us any good to stick our heads in the sand like that and pretend that the basic principles of opinion polling haven't been comprehensively tested and verified over a period of several decades. If you interview a genuinely representative sample of 1000 people and weight the results absolutely perfectly, then on 95 occasions out of 100 you'll get a result that is accurate to within a margin of error of 3%. That is a fact.

But that knowledge certainly doesn't mean that we should go to the other extreme and treat opinion polling as some kind of infallible God. The reason we should always approach polls with a critical eye is that they usually fall at least slightly short of the absolute methodological perfection that I've just described, meaning that on a substantial minority of occasions they'll be outside the standard margin of error. OK, it's very rare in western democracies for election results to vary dramatically from the final polls, but moderate divergences are much more common. The classic example in the UK was the 1992 general election, when an average of the final polls had Labour 1% in the lead, but the Tories ended up winning by 7%. About half of that divergence was caused by genuine methodological errors in the polls (the remainder was a late swing in opinion).

So probably the best attitude to take to any individual poll is : "This is likely to be vaguely in the right ball park, but it also has an entirely indeterminate 'real world' margin of error, because there's no way of knowing for certain how many methodological mistakes are being made, or how great the impact of those blunders are."

Furthermore, we're in a highly unusual position in this particular referendum campaign, because we already know for a fact that some of the polls are getting it wrong by a significant margin. The reason we know that is the huge and sustained divergence between the numbers being produced by different polling organisations. ICM, Survation and Panelbase are currently showing a Yes vote of 45-47% after Don't Knows are excluded, YouGov and TNS-BMRB are showing a Yes vote of 40-42%, and the extreme outliers Ipsos-Mori are showing a Yes vote of just 36%. There is simply no way of reconciling those figures, which means that one of two things must be true - either a) some of the polls are wrong to a substantial degree, or b) all of the polls are wrong to a substantial degree. The first possibility is obviously the more probable of the two, but the second can't be discounted - leaving open a small outside chance that Yes may already be in the lead (or indeed that No have a bigger lead than even Ipsos-Mori are reporting). To understand why, we have to look at the underlying reason for the highly unusual failure of the different pollsters to broadly agree with each other.

The methodology for standard voting intention polls is generally perfected by using the most recent election result as a baseline. If a pollster was out by 2% in that election, they'll ponder the reasons for that small error, and identify the tweaks they could have made that would have produced a totally accurate result. Implementing those tweaks in future polls significantly reduces the likelihood of reporting misleading numbers. But the problem is that the pollsters aren't actually able to go through that process for the referendum, because there is no baseline to work from. At best, we have the devolution referendum of 1997, but that took place in a different world - most of the present-day polling organisations weren't even around back then. So instead they've had to rely on hunches about which methodology will prove most accurate, and to some extent they've all been guessing differently. The chances are that at least one of them has guessed right, but it's just conceivable that none of them have. That's why there's a slightly greater risk of all the polls being substantially wrong in this case than there would be in a regular election. The lazy London-centric assumptions that have informed some of the guesswork don't help either, although that's become slightly less of a problem as time has gone on.

There's also another scenario in which Yes could be in a slight lead without that showing up in any of the polls, and it's one that has nothing really to do with methodological 'errors' as such. If supporters of independence are significantly less likely to reveal their true intentions to pollsters than opponents of independence are, then the Yes vote would be under-reported across the board. Intuitively that seems like a genuine possibility, but nobody really knows if it's going on. Pollsters can't realistically be expected to adjust their methodology to take account of a 'Shy Yes Syndrome' that hasn't yet been conclusively proved to even exist. And in any case, respondents would presumably be less shy in front of a computer screen than they would be when talking to a live interviewer, so the fact that four of the six active pollsters conduct their fieldwork online ought to be counteracting most of the reticence.

In a nutshell, then, although it's theoretically possible that Yes are already in the lead, there are no solid grounds that I know of for assuming that to be the case. The more plausible best-case scenario is that ICM, Panelbase and Survation are right, and that Yes are just a few percentage points behind with a bit of momentum in their favour. With five-and-a-half months still to go, that would be a pretty decent position to be in.

Unfortunately, we also have to consider the worst-case scenario, which is that Ipsos-Mori are right and everyone else is wrong. There's no getting away from it - the fact that the one and only telephone pollster in this campaign are producing such lowly figures for Yes is a cause for concern. It may mean that if telephone polling was the norm rather than the exception, the average No lead would be much higher than it currently is. But we simply don't know whether the telephone element of Ipsos-Mori's methodology is what's causing the difference, or whether it's something else entirely (such as the failure to weight by recalled 2011 vote). It's also perfectly possible that Ipsos-Mori are getting it wrong precisely because they are polling by telephone. It may no longer be feasible to reach a fully representative sample of the Scottish population in the way they are attempting. (For example : are they phoning landlines only? No-one seems to know.)

I was reading an interesting article by ICM's Martin Boon recently, about the tendency for UKIP to do much better in online polling than in telephone polling. He didn't offer any definitive answers about which was more likely to be accurate, but based on past experience he did raise the intriguing possibility that the results of different types of polls would "magically converge" when polling day came into view. Could that happen in the independence referendum as well? If so, it would suggest that all of the pollsters are telling us a "form of the truth" in a mysterious way that we don't fully understand, and that all of them are equally important. I don't really believe that, though - my guess is that Panelbase and Ipsos-Mori will still be miles apart in September, meaning that at least one of the two will be proved hopelessly wrong. If one of them is showing a Yes lead and the other is showing a No lead, we really will be in for a terrifying few hours.

But perhaps campaigners' experiences on the ground can provide clues about which pollster is more likely to be correct? To return to the 1992 example, Neil Kinnock famously suspected in advance that he was going to lose, and that foresight didn't come from private polling - it was from looking people in the eye on the campaign trail. So from that point of view, the fact that Yes canvassers seem to be so buoyant at present is certainly a cause for encouragement, but it's scarcely definitive.

OK, let's turn now to the specific points that Auld Acquaintance made in his blogpost -

"Why do I think the polls are wrong too?

The evidence is out there, it is increasingly clear that Better Together cannot get the numbers of people supporting them on the ground, the few poor souls they do get cast very lonely figures on their stalls and they cannot give away their wares, the public do not want to know. Yes stalls are full of support, and people queue up to chat, ignoring Better Together.

People in their droves across the country turn up for Yes events, a mere handful turn up for Better Together."

None of these factors strike me as being proof of anything at all. It may simply be that Yes voters are far more enthused, and that No voters just don't care about the referendum as much (or at least not yet). And perhaps people have more questions for the Yes side than for the No side.

"There are debates regularly taking place in schools, universities, village halls and consistently YES wins, by larger and larger margins, when was the last time you heard of NO winning one?"

Again, the attendees at debates are self-selecting, and if Yes voters are more excited by the referendum you'd expect a skew towards them. What is genuinely encouraging is the swing to Yes that often occurs between the 'before' and 'after' straw polls at the debates. That could be a portent of great things to come when the currently disinterested section of the population is exposed to saturation TV coverage of the referendum in August and September. But it isn't in itself a sound reason for thinking that the polls are wrong right now.

"Yes campaigners and Radical Independence have been doing their own polling, and the results are nearly all positive in the favour of YES."

I presume that simply refers to canvassing, and if so it's a positive sign, but one that has to be treated with extreme caution, because canvassers are sometimes told what they want to hear. At the Cowdenbeath by-election, the SNP claimed that they had found 41% of voters were planning to vote Yes, and 36% were planning to vote No. But Labour claimed that their canvassers had come up with numbers that bore no resemblance to that, with No at around 60% and Yes in the mid-teens. Are Labour lying to us? Or were people lying to Labour because they were embarrassed to tell their traditional party that they took a different view on independence? My gut feeling is that the SNP figures are at least somewhat closer to the truth, but it must be obvious that there's huge scope for uncertainty here.

"The polling companies by taking equal numbers from each of the class bands have this wrong.

If you take the AB’s as being NO, and the DE’s as being YES, on first sight that would seem a fair balance, would you not?

You would be wrong, because in Scottish society the actual numbers of people in class AB, are dwarfed by the numbers of people in classes DE."

The problem with this claim is that I just can't see any evidence that the polling companies are taking equal numbers from each class band. For example, the last YouGov poll had 354 respondents from the DE category after weighting, compared to just 209 from the AB category. We're not dealing with complete imbeciles here - you can be sure that the pollsters will be at least attempting to interview a representative sample of the Scottish population, based on specifically Scottish data from the census and other sources. That doesn't mean that they won't sometimes make mistakes - as I've mentioned several times, Panelbase had to change their methodology a few months ago because they were under-representing older voters. But that particular mistake was actually flattering the Yes campaign.

So all in all I think there's probably a fair bit of wishful thinking at play in Auld Acquaintance's post. It's true that I've been just as critical of the polls over the last couple of years, but that's been for completely different reasons, mainly relating to lack of transparency, biased preambles, and weighting by recalled vote from 2010 rather than 2011. Most but not all of those issues have now been resolved. I'm still troubled by Ipsos-Mori's extreme secrecy, by YouGov's bizarre practice of splitting 2011 SNP voters into two categories and weighting them separately, by TNS-BMRB's apparent presumption of a 100% turnout, and by the possibility that Panelbase may yet revert to their subtly biased preamble in future polls.

Whether we think the polls are right or wrong, though, one thing there isn't any doubt over is the trend. Just lately that's proved to be a relentlessly good news story for Yes.