In some quarters, ICM's Martin Boon has been widely praised for his piece in Scotland on Sunday explaining the thinking behind his company's approach to "indyref" polling methodology. It was certainly an interesting insight into thought-processes that normally remain private, although as I've already mentioned I find it extremely troubling that Boon seems to be taking seriously the very speculative notion that the Yes vote is being overstated due to a 'Shy No Syndrome', but doesn't even seem to have considered the much more plausible possibility that the No vote is being overstated due to 'Shy Yes Syndrome'. More broadly, there was a certain amount of ambiguity in the article's language, which has directly led to a few daft new myths doing the rounds.
Exhibit A comes from hardcore anti-independence Tory troll Carlotta Vance (who I am now permanently banned from setting straight over at her preferred haunt, the 'Stormfront Lite' site Political Betting) -
"If the polls say 'Yes' but Scotland votes 'No' - we probably know why already."
She then goes on to quote Boon talking about "lifting the power of SNP voter voices" by weighting according to 2011 vote recall, thus implying that she has (to her relief) identified the 'innocent explanation' for the No lead having collapsed to just 3% in the latest ICM poll. Except that doesn't work. There were in fact 288 people in ICM's sample who voted SNP in 2011 - and they were weighted DOWN to count as just 238 respondents in the final results. The respondents who were significantly upweighted were non-voters from 2011 - only 181 of them were interviewed, but they counted as 309 people in the final results. Crucially, among this group there was a 10-point No lead - not huge, but enough to mean that this aspect of the weighting procedure is helping the No side, not the Yes side.
The point being of course that Boon was actually explaining how weighting by past vote recall might flatter Yes UNLESS non-voters from 2011 are also correctly weighted, which is exactly what ICM are now doing. That caveat seems to have to got lost in translation, at least as far as certain PB Tories are concerned.
What those of us who care about polling accuracy should actually be concerned about is the apparent failure of the pollsters to weight by country of birth, which we know is an extremely strong predictor of referendum vote (with those born south of the border being far more likely to vote No). The place of birth of ICM's sample is as follows -
Scotland : 75%
England : 15%
Wales : 1%
Northern Ireland : 1%
Outside UK : 8%
ICM claim in the introduction to their datasets to use targets for their weighted data that are derived from the 2011 census, which as far as I can see must mean that they are failing to weight by country of birth, because according to the census results the correct numbers for the voting-age population are as follows -
Scotland : 81.5%
England : 9.6%
Wales : 0.4%
Northern Ireland : 0.8%
Outside UK : 7.6%
So unless I'm missing something, it seems absolutely clear that a heavily No-voting section of the population has been over-represented in ICM's sample by several percentage points, meaning that even the wafer-thin 3% No lead in the headline numbers may be exaggerated.
* * *
Note : I've updated this post with the exact census figures for over-16s, as provided by Scottish Skier in the comments section below. SS also estimates that the No lead in the ICM poll would be just 1% if weighting by country of birth had taken place.
Bingo we are just about there.
ReplyDeleteOnwards and upwards
CoB 16yr + in 2011.
ReplyDelete9.6% England
0.8% Northern Ireland
81.5% Scotland
0.4% Wales
0.5% Republic of Ireland
7.1% Other
I did a simple re-weight. Gives:
Yes 40.3% / No 41.3%
The difference is not huge, but then the ICM sample seemed quite high for Yes from England CoB (28%). Would be nice if the case, but the sample is small.
Also would give only 34% definitely voting No, which would be about right. TNS had 30%.
This ~1/3 is the same group of people who say an iScotland would be worse off economically in the SSAS (33% long term average). Doesn't mean they actually think that - many will say it because they just don't want indy as they feel British.
They are the ~1/3 status quo people in the three way Quo/Max/Indy split, typically with a strongly to fairly (eqS&B) British natID.
All No can rely on.
---
Yougov have the same problem, even MORI. Panelbase look closer based on a natID type Q in November.
2235 weighted sample of Yougov Scottish panel here with CoB asked:
http://d25d2506sfb94s.cloudfront.net/cumulus_uploads/document/5vbhf8v5ik/YG-Archive-131127-Migration-Observatory-EngWales-Scot.pdf
Well out. That's before they do the ridiculous 2010 weighting thing which doesn't correct what they are trying to correct at all as evidenced by the large up-weighting they have to do for people who actually admit to Labour 2010 / SNP 2011.
When I went through the tables I noticed that the vote for the 'South' area of Scotland was up-weighted significantly (49 to 117) as well, and this along with the NE was one of only two areas showing a significant no lead(26% to 49%), is this result consistent with results of previous polls in the South of Scotland? Because that up-weighting seems quite significant to me, particularly from the relatively small sample size it started with.
ReplyDeleteScottish Skier : Many thanks, I've updated the post with those numbers. I tried to track them down earlier today, but the census website is so difficult to navigate.
ReplyDeleteAnon : We know that the Borders and D&G are very heavily No, but that shouldn't be the case for the South region more broadly.
Yes Borders is slogging away, but if anyone can't find any more No voters to convert, we'd sure as hell welcome a hand here.
ReplyDeleteNae bother on the census data James. Agreed on the census 'Explorer' being something of a nightmare; took me quite a few attempts to get it to give me stuff I wanted and even now it seems more luck than anything if I get it to work. I'll send you my e-mail; feel free to ask if I can help with data gathering.
ReplyDeleteHat's off to Yes Borders. I got a Yes paper through my door the other week. Impressive as I'm miles from civilisation and probably not on most maps :-)
I have some idea where you live, so I can say that wasn't me. However, I'm finding places I didn't know existed and getting papers to them, including cottages at the end of two-mile-long dirt tracks. Others in the group are saying the same, with much banter about Cold Comfort Farm (though I have to say some places in the middle of sweet nowhere are little bits of heaven).
ReplyDeleteI've collated a recent copy of the unredacted electoral register with both Google maps and the OS map, and I think there are only two addresses in my six-mile-square area I can't pinpoint. I've got it all tabulated and plotted, the better to get round everywhere with the minimum of fuss.
I don't know how much spare time you have from all the calculations, but you could take "ownership" of a manageable area round your own home and undertake to get the papers to these places. Your locak group would certainly love you for it.
I don't know how many newspaper deliveries and casual conversations on the way that it takes to convert one person to Yes, but I'm doing as much of it as I can. (I note that an undecided-leaning-to-SNP I had a long chat with while leafleting the back of beyond in 2011 has now sprung up on a short-list of known strong supporters, which is heartening.)