Blair McDougall and his Abominable No-Men will be thanking their lucky stars today that YouGov are one of only two BPC pollsters that do not apply any turnout filter or weighting to their headline results. That's the one and only reason why YouGov have not just become the fourth pollster in a row to report an increase in support for Yes. Here is the result of today's poll when only respondents who are certain to vote are taken into account (that's the filter Ipsos-Mori use for their headline numbers) -
Should Scotland be an independent country? (excluding Don't Knows)
Yes 42% (+1)
No 58% (-1)
Should Scotland be an independent country? (not excluding Don't Knows)
Yes 39% (+1)
No 55% (n/c)
And here is the result when only respondents who give at least an 8/10 likelihood to vote are taken into account (that's the filter Panelbase use for their headline numbers) -
Should Scotland be an independent country? (excluding Don't Knows)
Yes 41% (n/c)
No 59% (n/c)
Should Scotland be an independent country? (not excluding Don't Knows)
Yes 39% (+1)
No 55% (+1)
In case you're wondering, the logic for applying what may look like quite extreme filters is that people generally overestimate their own likelihood to vote. Ipsos-Mori only include people who say they are certain to vote in their headline numbers because there has been such a strong correlation in the past between the numerical strength of that group and actual turnouts in elections. No fewer than 82% of the sample in today's YouGov poll say they are absolutely certain to vote, so unless you think the turnout in September is going to be ridiculously high, the voting intentions of those people may well be a better guide to the state of play than the reported headline numbers.
As pointed out by Scottish Skier on the earlier thread, there are a couple of oddities about this poll. The first is that Labour have a small lead in Holyrood voting intentions - that's in contrast to an SNP lead in the last YouGov poll, an SNP lead in actual votes cast in the European elections, and a very handsome SNP lead reported in the first post-Euro poll to be conducted by No-friendly firm Ipsos-Mori. So that might make us wonder if YouGov have ended up with a particularly "Nat-light" sample in this poll, in which case the referendum figures might also be slightly suspect.
But the most peculiar thing of all is the result of the supplementary question about the J K Rowling non-story, and how it will affect people's voting intentions. What normally happens with this sort of question is that the respondents who claim that they are more likely to vote Yes or No as a result of Event X or Y are for the most part people who were planning to vote in that way anyway. So given that slightly more people said that the Rowling garbage would make them more likely to vote No than said it would make them more likely to vote Yes, we should find an implied No lead that is either the same or greater than in the headline numbers. Instead, we find the polar opposite -
Does the abuse directed at J K Rowling make you more likely to vote YES in the referendum, more likely to vote NO, or does it make no difference?
More likely to vote Yes : 6%
No difference, I would have voted Yes anyway : 35%
TOTAL YES : 41%
More likely to vote No : 12%
No difference, I would have voted No anyway : 39%
TOTAL NO : 51%
With Don't Knows excluded, it works out as...
Yes 45%
No 55%
I'm struggling to interpret those numbers in any other way than as meaning that there are considerably more people in the YouGov sample thinking about voting Yes than are willing to say so when asked a more direct question.
There's one other reason for being a touch cautious about the headline numbers in this poll, and it's effectively the Survation problem in reverse. In many previous YouGov polls, 18-24 year olds (they STILL aren't bothering to interview 16 and 17 year olds!) have been the most Yes-friendly age group - for example, in the last poll Yes were only 1% behind in that group. But in this poll, young voters have swung in the other direction to such a wildly implausible extent that they're now the most No-friendly age group. The impact of that swing (which in reality is likely to be an illusion caused by random sampling variation) has been magnified because the 74 real respondents in the group have been upweighted to count as 123 'virtual' respondents. That isn't quite as extreme as the upweighting Survation had to do, but it's still plenty enough to have had a potentially distorting effect.
James - I remember hearing a bit of a stir a while back about Yes voters who have signed up for YouGov in the past year but after initially insinuating a tendency to be Yes supporters (by giving voting intention or admitting past support for SNP) have never been included in any referendum polling. I include myself in this bracket. I understand how allowing all newcomers to participate could distort results in favour of yes as we are more likely to be actively engaged than you average No voter. However could the discounting of these people on mass be the reason for SNP-lite polling figures?
ReplyDeleteMore generally how do pollsters choose respondents on a poll to poll basis? i.e. are some people likely to have participated in all a companies polls or would they use a totally fresh sample of people each time?
Callum : On the first point, I think it's Panelbase you're thinking of? I haven't heard anything about YouGov excluding newcomers. Survation have closed their panel altogether, although as far as I know the referendum wasn't the reason for that.
ReplyDeleteOn your second question, it should be relatively random, but it depends on whether you're in a group that is under-represented or over-represented in the panel. For example, Calum Findlay has been polled by Panelbase over and over again, and that's presumably because they don't have enough young people.
Thanks, I see. Maybe I'm confusing/combining two stories then. I definitely know of 3 or 4 folk who have signed up to YouGov and have never been polled on referendum but that could purely be circumstantial. We'd all be in 25+ age group which could be well represented I guess.
ReplyDeleteit's a sad day when we have to consider a n 82% turnout as ridiculously high rather than thinking that 'normal' turnouts are pathetically low. oh well...
ReplyDeleteI don't think 82% is ridiculously high - in fact it's probably not that far away from what we now anticipate. I just meant that anything a lot higher than that would still seem implausible.
ReplyDeleteOf course it's impossible to predict a turnout from an online survey anyway - Ipsos-Mori and TNS will be better guides on that score.