Monday, June 16, 2014

ICM poll : Yes vote at 46.4% after 'undecided leaners' are taken into account

The datasets for yesterday's ICM poll are now up, and just like last month they make for fascinating reading.  Most importantly, they show that undecided voters are breaking heavily for Yes -

Which way do you think you are most likely to vote, "Yes" or "No"? (Asked to undecided voters only)

Yes 27%
No 10%

That's a really striking finding, and it's considerably more important than it would have been in earlier ICM polls, because the firm introduced a huge methodological change two months ago that significantly increased the reported number of undecided voters. It's crucial to stress that this can't be taken as proof that undecided voters in general are breaking for Yes - different polls are showing conflicting evidence on that front, and the sample sizes are too small to draw firm conclusions anyway. But a better way of looking at it is that undecided voters who say Yes when pressed are on exactly the same spectrum as the people who say Yes on the original question - it's just a different level of certainty. For some reason Yes leaners were more likely than No leaners to say they were undecided on the original question in this particular poll, and that may just be a freakish occurrence that has helped to suppress the Yes vote on the headline numbers. When 'undecided leaners' are added into the main results (with the hard-core of true undecideds excluded), it's sufficient to increase the Yes vote by 1.2%, and of course to decrease the No vote by the same amount.

Should Scotland be an independent country? (Including undecided leaners, excluding other undecideds)

Yes 46.4%
No 53.6%

That's after turnout weighting is applied - yet another extremely recent ICM methodological innovation. Without turnout weighting the Yes vote is 0.2% higher - not a huge difference, but enough to tip the balance on the rounded numbers and make it Yes 47%, No 53%. In the interests of fairness, though, I should point out that turnout weighting is probably a very good idea, and has actually helped the Yes vote slightly on the reported headline numbers (and that was also the case last month).

But the biggest methodological change of all is undoubtedly acting to suppress the reported Yes vote. Starting with the Easter poll, ICM have been massively upweighting respondents who report that they didn't vote in the 2011 Holyrood election. In the case of this poll, 188 real respondents have been upweighted to count as 313 'virtual' respondents. Although that portion of the sample isn't ridiculously No-friendly by any means, it does have a somewhat bigger No lead than the rest of the sample, so the effect of the upweighting is obviously to give No a small boost.

Respondents who recall voting for any of the four major parties in 2011 are naturally weighted down to make way for the non-voters, and it's interesting to note that SNP voters are weighted down by almost exactly the same amount as Labour voters. Not much succour there for the theory that weighting by 2011 vote recall is leading to an artificially "Nat-heavy" sample, and thus increasing the reported Yes vote.

When the poll was published, I pointed out that it would be well worth checking whether ICM had once again under-represented male respondents and over-represented English-born respondents - both errors that have the inevitable effect of suppressing the reported Yes vote. The answer turns out to be "yes", and incredibly the degree of error is almost identical to last month. Just 45.2% of the sample are male, even though the 2011 census showed that 47.9% of the over-16 population are male. And 14.6% of the sample were born in England, even though the census showed that only 9.6% of the over-16 population were born there.

I've reweighted the numbers for the whole sample to the correct target figures for country of birth, and it increases the Yes vote from 45.1% to 46.3% (and of course decreases the No vote from 54.9% to 53.7%). Those figures take no account of undecided leaners or indeed of turnout weighting, but presumably the margin of increase would be in the same ball-park on each measure - so Yes might well be above 47% when undecided leaners are added in. (In fact there's probably enough information in the datasets to attempt that calculation, so if anyone has got three hours to spare...!)

The under-representation of men has a much more marginal effect, but as we saw last month that can sometimes still be enough to tip the balance on the rounded numbers used for publication.

As I've said before, the error on the country of birth is probably caused by 'institutional inertia'. I would imagine ICM are only asking where people were born to judge the extent to which English-born and Scottish-born respondents are diverging, and it hasn't seriously occurred to them to weight by those findings because it's not something they do as a matter of routine. But the gender error is much more baffling, because it must be something that ICM are doing deliberately. Why would they think that there are 10% more women than men in the adult population? Perhaps they reckon there's been a disproportionate influx of female immigrants to Scotland since the census was conducted? And even if that is their reasoning (doubtful), why do they claim to weight by target figures derived from the census? It just doesn't make any sense.

17 comments:

  1. Could the mismatch between weighting of Scot/Eng born voters and demographics be due to likelihood to vote? Scots-born voter demographic containing more people that are unlikely to turn out and vote?

    ReplyDelete
  2. No, the figures I've given are for the whole sample, ie. before turnout weighting is applied. I think what's going on is that there's some factor that leads volunteer online polling panels to have a disproportionate number of English-born people on their books - YouGov seem to have the same issue.

    ReplyDelete
  3. In actual fact, ICM aren't assuming there are 10% more women than men in the Scottish population - it's even more extreme than that. 54.9/45.1 works out at just over 21.7%!

    Their weighting figures suggest (by my back-of-an-envelope calculation) that there are more than six adult women walking around the streets of Scotland for every five men.

    [The census stat you cite suggests a much healthier ratio of around 12:13.]

    ReplyDelete
  4. I did a quick re-weight on CoB and gender including learners and I get 48% Yes / 52 No ex DK.

    ReplyDelete
  5. SS : Thanks very much. As a matter of interest, do you have any idea what the correct target figures should be for social class? I think I found the figures a few weeks ago, but I can't seem to track them down again.

    Calum Findlay pointed out the other night that different firms are using radically different weightings for social class (and to be fair to ICM, that's one example of them being more favourable to Yes than other firms).

    ReplyDelete
  6. Scotland's Census 2011 - National Records of Scotland
    Table QS611SC - Approximated social grade - Household Reference Persons (HRPs) aged 16 to 64
    All household reference persons aged 16 to 64

    Scotland
    1,765,009

    AB Higher and intermediate managerial/administrative/professional
    327,941

    C1 Supervisory, clerical, junior managerial/administrative/professional
    559,669

    C2 Skilled manual workers
    385,670

    DE Semi-skilled and unskilled manual workers; on state benefit, unemployed, lowest grade workers
    491,729

    There you go.

    ReplyDelete
  7. Also

    National Statistics Socio-economic Classification (NS-SeC)

    All people aged 16 to 74
    3970530

    1. Higher managerial, administrative and professional occupations: 1.1 Large employers and higher managerial occupations
    65396

    1. Higher managerial, administrative and professional occupations: 1.2 Higher professional occupations
    278832

    2. Lower managerial and professional occupations
    802545

    3. Intermediate occupations
    513038

    4. Small employers and own account workers
    294610

    5. Lower supervisory and technical occupations
    326930

    6. Semi-routine occupations
    616404

    7. Routine occupations
    514036

    8. Never worked and long-term unemployed: L14.1 Never worked
    124530

    8. Never worked and long-term unemployed: L14.2 Long-term unemployed
    74100

    L15 Full-time students
    360109

    ReplyDelete
  8. Also

    National Statistics Socio-economic Classification (NS-SeC)

    All people aged 16 to 74
    3970530

    1. Higher managerial, administrative and professional occupations: 1.1 Large employers and higher managerial occupations
    65396

    1. Higher managerial, administrative and professional occupations: 1.2 Higher professional occupations
    278832

    2. Lower managerial and professional occupations
    802545

    3. Intermediate occupations
    513038

    4. Small employers and own account workers
    294610

    5. Lower supervisory and technical occupations
    326930

    6. Semi-routine occupations
    616404

    7. Routine occupations
    514036

    8. Never worked and long-term unemployed: L14.1 Never worked
    124530

    8. Never worked and long-term unemployed: L14.2 Long-term unemployed
    74100

    L15 Full-time students
    360109

    ReplyDelete
  9. Thanks once again. I suppose the operative words are "household reference person", so the overall percentages would depend on whether working-class households have more people in them (which they probably do). No wonder the pollsters can't agree on the correct weightings.

    ReplyDelete
  10. Oops, not sure why the second post appeared twice.

    ReplyDelete
  11. Sorry, I posted that before seeing your other two comments!

    ReplyDelete
  12. RE the past vote / didn't vote thingy.

    It's fairly standard for e.g. 80% to give a voting intention then only just over 60% turn out or 50% in the case of 2011.

    I have concerns therefore about up weighting significantly those who say they didn't vote in 2011. What you are doing here is replacing people who generally don't vote and don't mind saying so with those who do follow things, express an intention but are lazy when it comes to elections.

    My feeling is that your past vote numbers should match those normally giving election VIs instead; these are fairly steady from poll to poll.

    ReplyDelete
  13. I see that Arch-Thatcherite and BBC employee Andrew Neil is crowing about a poll question purportedly showing scots want Trident to stay.

    Unlike some Yes supporters I'm fairly benign when it comes to polling and use trends myself to demonstrate shifts and notable movement in opinion.

    However, any polling question that supposedly shows a majority of scots in favour of Trident or keeping it, is, quite simply, full of shit. I don't care who the pollster is that is simply light-years away from the reality on the street, in the workplace or in the pub.

    Trident is deeply unpopular. End of story. Anyone who raises the question of Trident and weapons of mass destruction as they leaflet in the streets and on doorsteps fairly quickly finds that out. If a polling question claims otherwise you can safely disregard it. It's about as convincing as a polling question claiming Cammie or Clegg are very popular in scotland.



    ReplyDelete
  14. Mick Pork
    The Herald headline claims a "new poll", somewhere else says it's a year old. Who's right?

    ReplyDelete
  15. It's almost a year old - conducted between June and October of last year. It's only new in the sense that it's only just been published. It's also merely a subsample of a Britain-wide poll, and therefore of limited use.

    More to the point, it actually shows that Scots are opposed to nuclear weapons by 46% to 37% - almost exactly in line with an Ashcroft poll last year, which was also bizarrely misreported as "Scots are pro-Trident". In both cases, the question that generates such merriment among the Brit Nats is one that implicitly presupposes that Trident is going to be retained against our wishes, and asks for a practical view on where it should be based.

    ReplyDelete
  16. Had to laugh at BT Facebook post, making a big deal of a comres poll for south of Scotland only, turned out 70 to 30 for No.

    Claiming this show momentum with No!!

    Really clutching at straws and BT supporters taking this as proof they are winning all over Scotland.

    ReplyDelete
  17. I've felt for a long time that the accredited (but not credible) polls have got it badly wrong and your latest post and comments gives a few pointers why.
    1. In the early days activists mainly English in the BT mob applied to and have swamped the online panels. This is now turning up in the poll results but is not being 'corrected' properly by weighting.
    2. The pollsters don't want to change the errors about females because those help the NO case. Even Pollsters have to eat and as their employers are mainly on the NO side, they are not going to change.
    3. In my view you doth grant the pollsters too much integrity. They are businesses after all which need customers. And being YES favourable is not a good business option for pollsters in the Indy campaign. Compare the numbers of Panelbase polls and others; and even Panelbase is wriggling.


    ReplyDelete