A bit more on polling methodology

I thought you might be interested in this comment left by a user on Reddit. (I stumbled across it because someone was kind enough to link to Scot Goes Pop on the same thread!)

"Met someone last week who until the end of last year worked for Mori. Asked him about the polling situation and he said there's a general understanding among pollsters that most of the weightings used have been badly chosen or chosen specifically in some cases, or not at all in others where there should have been some form of weighting. On the reason why, well he was a little bit coy about this so no insight gleaned here.

I wasn't personally aware of such a thing as poll weighting and for example that some polls last year showing a large No lead were weighted towards Westminster voting intentions, even though the focus of the poll was clearly the independence referendum.

I asked about the likely accuracy of SNP door to door canvassing data collected among the local by-elections and in general. He said it was understood to be pretty good, I asked if 40-50% support for independence was reasonable in this case, he said yep. I asked how much the polling from likes of his recent employers and other large pollsters in the UK could be out based on his observations about the weighting. 10-20% he said."

Of course we have to be cautious about this, because the catastrophic error of weighting by recalled Westminster vote has now been largely resolved (touch wood), so that should no longer be responsible for distorting the numbers. That said, YouGov are clearly still in denial about the sheer scale of the methodological change that they introduced back in September. A day or two ago, Peter Kellner wrote the same bloody article he's been periodically writing for the last three years about how Scottish public opinion on independence is supposedly set in stone - well, how the hell does he square that claim with the massive 10% drop in the No lead that his own firm reported in September? Presumably we're meant to infer that the shift in opinion was not real, but an illusion caused by the methodological change - but has anyone actually heard Kellner say that directly? I certainly haven't, and no wonder - it would be tantamount to an acknowledgement that YouGov had previously been pumping out grossly distorted figures for months, if not years.

To return to the original point, though, it's interesting to hear some anecdotal evidence that pollsters themselves are privately anxious that they may have been understating the Yes vote, and perhaps still are. As far as canvass returns are concerned, they do come attached to a health warning, because there will always be some people who tell the canvasser what they think he/she wants to hear. But experienced people in the Yes campaign will be aware of that problem, and will be able to make a common sense adjustment to the numbers to take account of it. If the pattern still bears no resemblance to the polls even after that, it could be an indication that at least some of the pollsters are indeed getting their methodology wrong. But I can only speculate on what the canvass figures are actually showing - the only proper insight we've had was the well-publicised Cowdenbeath example.

I also had this little exchange with Duncan Hotdogstall (no relation) on the previous thread about the new Ipsos-Mori poll -

Duncan Hotdogstall : Hi James,

A glance through the data tables is setting off some alarm bells. I'm starting to wonder if I am looking at the wrong tables? Can you advise?

On page 2 the table says "Party support Scottish Parliament Holyrood First Vote Constituency" Does this mean it is weighted to Holyrood? If so then the weighted SNP vote appears to add up to 36% (302/845). We all know the actual, real life vote was 45.4%.

The sample also seems to be less than representative of the general population when it comes to national identity. The chart of page 4 of the tables says that a total of 48% of the sample feel either Scottish only or more Scottish than British. Yet the most recent census data tells us that 62% of the population feel Scottish only. Conversely, 48% of the sample feel more British or British only, whereas the census data gives that figure as around 34%.

Finally, the sample and census data depart again when it comes to place of birth: On page 5 of the chart it says that 78% of the sample were born in Scotland. The census says the figure is 83%.

Therefore, it appears that the sample in this poll is far more "British" than you would find in the usual population and way less representative in terms of SNP voters.

What is your analysis, James? Scottish Skier seemed to think that the result, when adjusted, should be more like Y 37, N 43 and DK 20, but the explanation wasn't very detailed.

Me : The Holyrood figures are current voting intention, not recalled 2011 vote.

It's not really possible to make a direct comparison with the national identity figures in the census, because the census asked the question in a different way. But even comparing to the most recent SSAS, Ipsos-Mori's Scottish figures are a bit too low.

As Oldnat pointed out the other week, the problem with making a comparison with the census figures on place of birth is that the census included children. I've tried to find the figures for 16+ only, but without any luck so far.

I think what Scottish Skier does is take an average of the SSAS national identity figures over the last fifteen years, and adjust according to that. I'm not sure that's wise, because there's fairly clear evidence that Scottish identity has fallen back on the SSAS in recent years. So he's probably making too radical an adjustment.

Duncan Hotdogstall : Thanks James. Although I still can't believe that poll, no one in the real world is experiencing what it claims. I personally know 5 people who have moved to yes in the last month (including a tory, oh yes, he made a right song and dance about it on facebook) but have yet to meet a single person in the world ever who has moved from yes to no.

Sorry for the double post earlier, don't know what happened.

Me : OK, I've done a rough calculation to reweight the Ipsos-Mori figures in line with last year's SSAS national identity data, and it comes out as Yes 32%, No 51%. That isn't directly comparable to the published headline figures because it isn't filtered by certainty to vote, but essentially it has reduced the No lead by about 7%. Not quite as dramatic as what Scottish Skier came up with, but certainly an indication that Ipsos-Mori should be taken with a pinch of salt.