ICM poll : Yes vote at 46.4% after 'undecided leaners' are taken into account

The datasets for yesterday's ICM poll are now up, and just like last month they make for fascinating reading.  Most importantly, they show that undecided voters are breaking heavily for Yes -

Which way do you think you are most likely to vote, "Yes" or "No"? (Asked to undecided voters only)

Yes 27%
No 10%

That's a really striking finding, and it's considerably more important than it would have been in earlier ICM polls, because the firm introduced a huge methodological change two months ago that significantly increased the reported number of undecided voters. It's crucial to stress that this can't be taken as proof that undecided voters in general are breaking for Yes - different polls are showing conflicting evidence on that front, and the sample sizes are too small to draw firm conclusions anyway. But a better way of looking at it is that undecided voters who say Yes when pressed are on exactly the same spectrum as the people who say Yes on the original question - it's just a different level of certainty. For some reason Yes leaners were more likely than No leaners to say they were undecided on the original question in this particular poll, and that may just be a freakish occurrence that has helped to suppress the Yes vote on the headline numbers. When 'undecided leaners' are added into the main results (with the hard-core of true undecideds excluded), it's sufficient to increase the Yes vote by 1.2%, and of course to decrease the No vote by the same amount.

Should Scotland be an independent country? (Including undecided leaners, excluding other undecideds)

Yes 46.4%
No 53.6%

That's after turnout weighting is applied - yet another extremely recent ICM methodological innovation. Without turnout weighting the Yes vote is 0.2% higher - not a huge difference, but enough to tip the balance on the rounded numbers and make it Yes 47%, No 53%. In the interests of fairness, though, I should point out that turnout weighting is probably a very good idea, and has actually helped the Yes vote slightly on the reported headline numbers (and that was also the case last month).

But the biggest methodological change of all is undoubtedly acting to suppress the reported Yes vote. Starting with the Easter poll, ICM have been massively upweighting respondents who report that they didn't vote in the 2011 Holyrood election. In the case of this poll, 188 real respondents have been upweighted to count as 313 'virtual' respondents. Although that portion of the sample isn't ridiculously No-friendly by any means, it does have a somewhat bigger No lead than the rest of the sample, so the effect of the upweighting is obviously to give No a small boost.

Respondents who recall voting for any of the four major parties in 2011 are naturally weighted down to make way for the non-voters, and it's interesting to note that SNP voters are weighted down by almost exactly the same amount as Labour voters. Not much succour there for the theory that weighting by 2011 vote recall is leading to an artificially "Nat-heavy" sample, and thus increasing the reported Yes vote.

When the poll was published, I pointed out that it would be well worth checking whether ICM had once again under-represented male respondents and over-represented English-born respondents - both errors that have the inevitable effect of suppressing the reported Yes vote. The answer turns out to be "yes", and incredibly the degree of error is almost identical to last month. Just 45.2% of the sample are male, even though the 2011 census showed that 47.9% of the over-16 population are male. And 14.6% of the sample were born in England, even though the census showed that only 9.6% of the over-16 population were born there.

I've reweighted the numbers for the whole sample to the correct target figures for country of birth, and it increases the Yes vote from 45.1% to 46.3% (and of course decreases the No vote from 54.9% to 53.7%). Those figures take no account of undecided leaners or indeed of turnout weighting, but presumably the margin of increase would be in the same ball-park on each measure - so Yes might well be above 47% when undecided leaners are added in. (In fact there's probably enough information in the datasets to attempt that calculation, so if anyone has got three hours to spare...!)

The under-representation of men has a much more marginal effect, but as we saw last month that can sometimes still be enough to tip the balance on the rounded numbers used for publication.

As I've said before, the error on the country of birth is probably caused by 'institutional inertia'. I would imagine ICM are only asking where people were born to judge the extent to which English-born and Scottish-born respondents are diverging, and it hasn't seriously occurred to them to weight by those findings because it's not something they do as a matter of routine. But the gender error is much more baffling, because it must be something that ICM are doing deliberately. Why would they think that there are 10% more women than men in the adult population? Perhaps they reckon there's been a disproportionate influx of female immigrants to Scotland since the census was conducted? And even if that is their reasoning (doubtful), why do they claim to weight by target figures derived from the census? It just doesn't make any sense.