Over the last day or two, you might have seen the referendum data from May/June published by the British Election Study. It shows a relatively narrow (albeit hardly untypical) No lead of 51% to 39%, which is down from a 52% to 37% split in February/March. But what I didn't realise initially was that the fieldwork was conducted by YouGov, whose panel is notorious for producing higher No leads than the average (both in polls for YouGov themselves, and in Progressive Partnership polls for which YouGov conducted the fieldwork). There has been no official YouGov poll so far in this campaign that has shown a No lead of as low as 12% (the lowest was 14%), and nor has there been one that has shown a Yes vote of as high as 39% (the highest was 37%). Yet the unusually narrow gap in the BES data has been found in spite of the sample size being five times bigger than for a standard YouGov poll.
Overnight, I was sent the following commentary on the survey results by someone who said "no need to mention me" - I'm not sure whether that was a specific request for anonymity or not, but to be on the safe side I'll assume that it was! Incidentally, I've changed the Wave 1 Yes figure below from 38.6% to 36.6%, which I'm pretty sure is the correct number.
I am a YES supporter and canvass in Edinburgh. I am not an expert on opinion poll methodology but analyse statistical data from scientific experiments so have some familiarity with issues involved in estimation.
I thought I would pass on some information about the "British Election Study" that was released today. I was at the academic launch meeting in Edinburgh today.
Some detail on methodology
The data was collected **online** by **YouGov** on behalf of BES. The Wave 1 (i.e. 20 Feb to 09 March) sample was N=5896 and Wave 2 (i.e. 22 May to 18 June) was N=6182. The percentages for Wave 1 was 52.0% NO, 36.6% YES and 11.4% undecided. Wave 2 was 51.0% NO, 38.8% YES and undecideds 10.2%. I copied these figures from the powerpoint presentation of Prof Ed Fieldhouse.
Although it can be classified, with regards the IndyRef, as a YouGov poll, many other questions were added (over 200, on a wide range of topics). These 6000 Scots voters were part of a larger sample that had 20,000 English and 3,000 Welsh. The Scottish referendum question was not asked upfront therefore. Also, as part of a GB poll, it did not include voters who were 16 and 17.
Prof Curtice chaired the meeting and commented that, for a YouGov poll, it was not as NO-friendly as expected.
I have some other hand-written notes but I hope this helps to put it in context.
NOTE for OPINION POLL GEEKS (like me :-)
The BES polling data was collected by YOUGOV on behalf of BES.
This poll should thus be compared to recent YouGov IndyRef polls. YouGov is one of the 6 polling companies charting the progress of the referendum, and tends to produce high leads for the NO camp. Polls have shown NO leads ranging from 3% (Panelbase) to 19% (YouGov) recently.
These results above show NO at 51.0%,YES at 38.8% and UNDECIDED at 10.2% (data collected 22 May to 18 June, sample size 6162) at the later date and NO 52.0%, YES 36.6%, UNDECIDED 11.4% (data collected 20 Feb to 09 March), sample size N-5896) at the earlier date.
Note also that 16 and 17 year-olds were not included.
COMPARISON with YouGov polls
These N=6000 polls can be compared with YouGov polls with N=1200 samples. However, note that these larger polls ask other questions (over 200 questions) and the Scottish IndyRef question is not asked upfront. Note also that 16-17 year olds were not included in the N=6000 polls.
N=6000 polls and N-1250 YouGov polls
These two snapshots in Feb/March and May/June have NO leads of 15.4% and 12.2% respectively, based on samples of 6000 voters each.
Standard Indyref polls with sample sizes of about 1200 in approximately the same period showed a NO lead of 18% (24-28 March, N=1257) and a NO lead of 19% (25-29 June, N=1206).
* * *
One point I should add for clarity is that YouGov never interview 16 and 17 year olds, even for dedicated referendum polls - they're the only pollster that still fails to do so.
If You Gov do not interview 16 or 17 year olds, then their polls are more worthless than their normal samples/weightings.
ReplyDeleteThe way Yougov, and I assume other panels, works is that panel members get a drip of short surveys. Most of the questions are about what do you think of brands and other consumer stuff. From memory the indyref questions are prominent. I assume that the crticisms of Yougov panel data in the referendum are less valid because BES are quite capable of selecting the sub-panel and weighting the data themselves. I say "assume" because it is not clear to me how the panel was constructed and the data weighted. The sample size and the fact it was analysed by respectable academics did seem to tick the odds to 6/1 and 1/6, whereas the online market has been slow to react to other polls.
ReplyDeleteShale Bing : Are you suggesting that the odds on Yes lengthened as a direct result of this study being released? I find that hard to believe, but if so, it's totally and utterly irrational. Like you, I'm unclear on whether YouGov's own weighting scheme was used, or whether the BES did the weighting themselves, but either way, it's startling to see the Yes vote as high as 39% in data collected from a panel which is known to be extremely No-friendly. (As John Curtice has pointed out, YouGov show a lower Yes vote even among Labour voters, who aren't affected by the "Kellner Correction".)
ReplyDeleteSo if anything the release of this data should have led to the Yes price shortening, albeit the fieldwork is several weeks out of date.
I prepared this for the previous article, but here's ok I imagine...
ReplyDeleteOn the topic of CoB, NatID etc…
MORI are the only one to ask a NatID type Q consistently...
What’s fascinating is that since MORI have been asking NatID, the number of British ID’ing (equally S&B + more B than S + B not S) people in Scotland has been steadily rising, whilst the number of people ID’ing as Scots (S not B + More S than B) has been falling.
We’ve gone from 54 S / 43 B in mid 2012 to 48 S / 49 B in May. Scotland is now more British than ever according to MORI, going both against the census and the SSAS.
Well, it’s really unheard of that people change their national identity on such short timescales, but maybe it’s because No is heading to victory, they have been consistently building up a huge lead and everyone is becoming more British. The census and SSAS are just wrong too.
Except there’s a wee problem with this. They’re becoming more British yet more supportive of independence for Scotland according to MORI. For example, the No lead almost halved from 29 to 18 points from May 2013 to May 2014.
There is an alternative explanation that makes more sense. Landline telephone polls have a ~20% response rate now. You don’t have to agree to take the poll which is ‘about the referendum’ (you are told what it’s about before they start and asked whether you wish to proceed – I’ve been polled by them). What would be more plausible is that fewer and fewer Scots (who are much more likely to be Yes) have been agreeing give their views, making the base increasingly British (more No) in nature. At the same time, those more British who have been answering, have been moving to independence.
I wonder what the results would look like if the response rate was better in terms of Scottish NatIDers. Quite possibly very scary for the pro-union side. Yes ahead is perfectly plausible.
There’s evidence it’s happening in online ICM and panelbase too; Scottish respondents falling. We don’t have enough data here, but the consistent over representation people from England with not enough Scots backs things up. Same of course for Yougov.
Do (more No) people from England just like answering polls more? Or is it that shy Yes Scots are just more likely to not answer polls at all. We can be 90% sure shy Yes exists; ICM just gave us the best evidence yet – those who said No then marked themselves as a Yes on the 1-10. Effectively no Yes voters made a similar ‘mistake’.
This may go some way to explain why Yes is ahead from canvassing, yet still a bit behind in polls. It is the nature of the situation; Scotland is about to do something huge, something it is being screamed at for in the most abusive and horrible manner from just about every corner. What do people expect to happen, but for the average person to just keep their head down then quietly tick their box of choice on the day which will bring it all to an end (that’s Yes BTW, as a No just means more years of abuse with Tory rule fringe benefits).
I'm pretty sure yesterday Blair McDougall was tweeting triumphantly that the BES poll was showing an INCREASED lead for No between March and June. Mysteriously those tweets seem to have disappeared!
ReplyDelete@Theuniondivvie: Yes, I saw that tweet from Blair McDougall yesterday, it has definitely disappeared, this appears to be the only reference that remains:
ReplyDeletehttps://twitter.com/ORScotland/status/493688867505971200