A guest post by Alasdair Stirling
YouGov's Peter Kellner has written an extraordinary blog article entitled 'Why Do the Polls in Scotland Vary So Much?' in which he puts forward what Sir Humphrey might call a 'brave' proposition. The gist of Kellner’s view is that the Scottish independence referendum opinion polls (other than YouGov's of course) fail to capture the true character of the constituency of Scottish voters that supported the SNP in 2011. He contends that, absent a corrective methodology such as YouGov applies, polling samples will contain too many of what he terms 'Passionate Nats' and too few – again his term - 'Passing Nats' and that that the resulting imbalance skews the forecast Yes/No percentages in favour of Yes to independence.
Now the SNP is not unique in finding its support from both committed loyalists and floating voters; all political parties experience this phenomenon so it might come as a surprise that Kellner does not advocate any similar corrections for poll respondents that supported the other Scottish political parties in 2011. It is not hard to see a rationale for this approach as it may be arguable that changes in voter allegiance amongst the Unionist parties will have little effect on the the binary Yes/No independence voting intention.
However, Kellner is in some difficulty with his policy of correcting only the poll respondents supporting the SNP in 2011 when it comes to forecasts of party support vis-a-vis the 2016 Scottish general election. Clearly if all of the Scottish parties are susceptible to this problem but the correction is applied to only the SNP supporting respondents, then it is likely that the forecast levels of party support for the 2016 election will be wildly inaccurate. Accordingly, by deduction we have to conclude that Kellner believes either: that the problem applies only to the respondents that supported the SNP in 2011; or that in so far as it applies to all parties its effect is only statistically significant as regards respondents that supported the SNP.
Kellner uncovers this phenomenon – lets us call it the 'Kellner Effect' – by investigating the voting record of 2011 SNP supporting respondents and cross-referencing the same respondents' 2010 Westminster election voting record (YouGov of course rely more on their panel data than voter recall). Kellner advocates that SNP supporting poll respondents be disaggregated into two constituent groups (loyalists and floaters or in his terms 'Passionate' and 'Passing') sampled according to quotas established by reference to the 2010 election and then re-aggregated by means of (if necessary, aggressive) political weighting. Kellner, without explanation, limits the application of this methodology – lets call it the 'Kellner Correction' – to SNP loyalists and Labour floating/tactical voters and concludes his blog article with the assertion that any poll that does not apply the Kellner Correction is wrong (and we must suppose just so much toilet paper).
In his UK Polling Report blog Anthony Wells gives a fine explanation of the sampling and weighting methodologies used by the polling industry. In regard to sampling, he points out why a truly random sample is impractical and cogently explains the Quasi-random and Quota sampling techniques that the polling industry uses to achieve viable polling samples before going on to explain the use of weighting to correct discrepancies and bring a viable polling sample more completely into line with the nature and character of the broader community being sampled.
Quasi-random and Quota sampling techniques are therefore the very foundation stones of the modern polling industry and the Kellner Correction can only be necessary if normal Quasi-random and Quota sampling techniques are unable to capture the true character of the community of SNP voters in Scotland. Absent some malign manipulation of the polling process, which Kellner does not suggest or imply, we must take his blog article as challenging the foundations of political polling. For if the Kellner Effect can apply to SNP respondents in Scotland, why not Conservative/Labour/Liberal/UKIP respondents throughout the UK; and if so, where when and why? Can we reasonably consider the merits of any poll (certainly any Scottish poll) that does not rigorously apply the Kellner Correction?
However these considerations are just the start. The broader implications of Kellner’s Effect and Correction are nothing short of staggering! If Kellner is right, then any polling company not applying the Kellner Correction can effectively shut up shop (at least in Scotland) and YouGov are positioned to establish a monopoly of political polling in a revalidated United Kingdom whilst the rest of the industry plays catch-up. On the other hand, if he is wrong it is hard to see that Kellner himself (having allowed his company to depart so fundamentally from the industry’s tried and tested methods) can survive an independence referendum result that falls into line with the forecasts of the polling companies not employing the Kellner Correction. But beyond potentially ruining his own career, Kellner’s is also throwing the dice for YouGov itself as we can only contemplate the detrimental effect on its business if events prove to its customers that YouGov lacks a commitment to the industry’s tried and tested methods, impartial approach and rigorous standards.
Notwithstanding, perhaps the greater implications lie in the referendum debate itself. With other polling companies showing a narrowing of the forecast Yes/No result, the Better Together campaign take much succour from YouGov’s consistently strong forecast of an emphatic No vote (and its effect on the many Poll of Polls type averages). Better Together’s Campaign Director Blair McDougall (not to mention Labour activist Ian Smart) have much tongue-in-cheek fun tweeting ‘no room for complacency’, but if Kellner is wrong then Better Together really do have no room for complacency, but will thanks to Kellner remain blind to the need to alter strategy. So in addition to imperilling his own career and business, Kellner may yet turn out to be massively assisting Yes Scotland in bringing about the end of the United Kingdom.
Wow, great article. Kellner is on a mission, but it will end in disaster.
ReplyDeleteWhat are the comments from the rest of the industry? Any thoughts from Prof Curtice?
Thanks for an interesting bit of analysis. The other question I would have for all the pollsters is how do they weight for people who did not vote in 2010 or 2011? The polls and the increased voter registration statistics see to point towards a large number of non-habitual voters on 18th September. And anyone aged 16-20/21 did not vote in 2010.
ReplyDeleteIn addition to Andrews comment, if they weight based on voting response given just after the 2011 election rather than vote recall then how do they handle panel members who joined after 2010/11?
ReplyDeleteCould this mean a panel largely made up of old guard YouGov members as opposed to new sign ups who are not trusted to recall voting intention?
As I said before, my neighbour is one of the Liberal 2010 + SNP 2011 voters who's raging that Kellner doesn't give him his own wee group to be in.
ReplyDeleteI voted SNP + Margo 2007, then SNP + SNP in 2011 after moving to the Borders. Do you think there's scope for a grouping for me? I remember what I voted in 2010 if that helps.
I guess the main hurdle I'll have to first overcome is being born in Scotland. That does reduce your chances of being in a Yougov Scottish panel somewhat.
Yes, I've been wondering about that. I have a feeling that new sign-ups are asked for their vote recall as soon as they join, as opposed to them being asked in each individual poll. So it may be a mixture of old and new, in which case Kellner's approach has its limitations (even assuming it's not entirely misconceived in the first place).
ReplyDeleteMy comment was a response to Callum, by the way!
ReplyDeleteI think it should be called the 'Brigadoon Correction' due to its ability to accurately replicate the views of Scottish society as seen from London.
ReplyDeleteNice article James.
Thanks, but I should emphasise that it was written by Alasdair Stirling!
ReplyDeleteI meant how you presented it on your website of course James. Lovely. Ahem. ;-)
ReplyDeleteExcellent stuff Alisdair and nice angle to take.
While I've joked quite a bit about all the wonderful groups you could create (SNP 1979 + Labour 79-97 + Liberal 01-05 + SNP 2010 is a possibility for example), I hadn't thought about how this is going against the whole basic fundamentals of polling even though I sort of knew that if you catch my drift.
It's interesting to note that Rev Stu the scourge of anything that is decent about Britain is in fact a Liberal voter!
ReplyDeleteExcellent article from Alasdair Stirling and thanks to James as these guest pieces are always nice to see.
ReplyDeleteSurvation have responded in detail on their blog.
http://survation.com/response-to-yesterdays-times-yougov-articles-and-yougovs-published-research-about-survations-scottish-independence-methodology/
Survation @Survation · 5h
Response to @thetimes, @YouGov articles and YouGov’s published research about Survation’s #indyref methodology: http://survation.com/?p=5750
Well worth looking at and I would be very intersted to hear the thoughts of James, Alasdair, Scottish_skier and other on it.
In particular James, is this true?
"That said, the practice of commissioning your own research solely for the purpose of criticizing another polling company’s methodology seems, in our view, to be unprecedented in the industry and not something that we would wish to repeat."
I must admit that's a part of Kellners extraordinary outburst that I missed, but was it indeed the case?
If so then Kellner really has lost it completely and gone off the deep end. John Curtice had better find some excuses very quickly for his feeble silence and inaction as all this plays out.
Yes, it was pretty clear from what Kellner wrote that his research was prompted by the opportunity to make a comparison with Survation's own exercise, and it's very hard to believe that he didn't have this diatribe in mind when he came up with the idea.
ReplyDeleteTo be fair to Professor Curtice, a big part of Survation's counterargument hinges on a very sensible piece of advice that Curtice gave them at the start of the year (ie. that 2011 vote recall is much more accurate than 2010 vote recall).
James, I think you might be right. I am one of those more recent sign-ups and I do think I got asked up front how I voted in 2010/11. I am yet to be involved in an indyref poll however.
ReplyDeleteAlso one of those Lib/SNP switchers they fail to recognise...
I appreciate that this is a really good article for folks who have a good understanding of polls but I'm a newbie and don't really get what you are all on about.
ReplyDeleteIs Kellner saying that everyone who voted Labour in 2010 is a committed Labour supporter who then went to SNP in 2011 for tactical reasons? (To stop labour getting in for example).
Just to be clear, I have no wish to immediately move on from Alasdair's excellent article which deserves further discussion. I only bring Survation's response on their blog to the attention of James and others for future consideration and discussion (perhaps James could include some of it's conclusions in a future article) as well as just more info that helps inform Alasdair's excellent response.
ReplyDeleteThe first response on this thread is extremely pertinent as John Curtice is the current President of the British Polling Council.
So I'm afraid his silence and inaction speaks volumes as Survation is forced to defend itself from Kellner's eccentric attack.
Alasdair is absolutely correct in drawing attention to just how much of a radical departure from the rest of the BPC Kellner and YouGOv have committed themselves to for the Independence referendum. This is no small matter but a BPC member going right out on a limb for reasons we can only guess at. We can also only guess at why John Curtice has allowed this to happen and seems clueless as to how to respond.
If Kellner's aim was to change industry practice then he should perhaps have done a far better job of justifying it than he has to date.
Though to be honest it's pretty hard to escape the conclusion that Kellner is just desperately trying to dig himself out of a hole as more and more people were asking serious questions about his bizarre methodology. Kellner's response has been to take the shovel and try to bash Survation and every other polling company about the head with it.
Given Kellner's many past examples of blundering into party politics and prejudging issues the wisdom of that approach looks more questionable than ever. To say the least.
Saw this?
ReplyDeletehttp://www.marketingmagazine.co.uk/article/1298311/better-together-recruits-david-camerons-former-strategy-director-andrew-cooper
Saying that the private polling presented to a high level Better Together meeting showed one third and rising for Yes, one third and FALLING for No! and one third DK.
Rob Shorthouse said it was garbage, but what can a man do when he needs to pay his mortgage?
Mike I had not seen the Survation response which was (as is to be expected) a forensic dissection of Kellner's argument. I was merely pointing out just how nutty Kellner's position is and the dangers for himself and YouGov.
ReplyDeleteThe thought that Kellner was Yes Scotland's hidden asset emerged in the first draft as mischievous dig at Better Together, but it found its way into the final text because the more that I thought about it, the more seemed to ring true.
As to Better Together's private polling, another interesting omission is the absence of any Better Together tweets or Facebook posts about their canvass returns. They are all over school votes in favour like a rash, but not a single tweet as to their successful canvass returns - we on the other hand are tweeting prolifically.
ReplyDeleteWe are all colored by current thinking in our recollection of past actions and I think polling companies are grossly overstating the accuracy of recollection. If you put together a sample that you expect to be representative based on age and education etc but end up with a skewed recollection of past votes, it seems to be a real possibility that recollection is altered by the current opinions. If that is the case then the opinion for independence is stronger than polling suggests. We will find out on September 19.
ReplyDeleteAlasdair your article stands very well on it's own and I was only bringing the Survation response to wider attention since it has a direct bearing on these discussions.
ReplyDeleteI agree entirely that Kellner and YouGov's actions are looking ever more bizarre and increasingly unjustifiable. Particularly for a supposedly serious pollster and member of the BPC.
Your points about Kellner and YouGov's actions backfiring and being completely counter-productive are very well made. The fact is YouGov were and still are the goto pollster for the No campaign and they were putting ever more weight on their findings. Now that all this has blown up in Kellner's face I can see no other outcome than a massive question mark against all YouGov referendum polling from now on as well as an intensive focus on all those reasonable methodology questions YouGov are still refusing to answer.
How could it be otherwise? Kellner and YouGov are quite obviously attacking any other pollster who does not use it's controversial methodology and "Kellner Correction". Kellner has gone all in on his new methodology so does he limit it to referendum polling or is he going to extend and adapt his "Kellner Correction" to all other YouGov polling? If it's such a superior a method then why wouldn't he?
All this controversy leaves the No campaign with a very stark choice. They either continue to believe and trumpet No friendly findings like the ones produced by YouGov and Kellner or they have to accept that the referendum is far closer than they are willing to admit.
I'm fairly sure No will continue to believe YouGov and bet everything on the most No friendly pollster being right. It's not as if they can do much about it anyway if the "Kellner Correction" is compete bollocks.
The No campaign can't go anywhere else. They are about as negative as it's possible to be right now while their 'secret weapons' (Harry Potter, Obama, Clinton, the Pope) have come and gone with no effect other that YouGov's small MOE change poll with it's eccentric methodology.
Considering the President of the BPC John Curtice was using that recent YouGov/Kellner polling to predict 'doom for the Yes Campaign' (yet again) this extraordinary controversy could claim far more victims than just Kellner and YouGov's credibility before it's over.
The real question here is why YouGov polls so consistently under-represent SNP Voters in the first place. That is what Kellner is complaining about after all. If he didn't have to re-weight the SNP tranche up-the-way he wouldn't have an embarassing Yes bulge. The big problem is that 44.7% voted for the SNP in the regional vote in 2011. Even at the time, YouGov consistently underestimated SNP support by a percentage in the teens. No "late SNP surge" type of argument suffices as the final YouGov poll in early May 2011 still put the SNP only a few points ahead, not the nearly 18 point lead in the election result.
ReplyDeleteThe YouGov data are apparently as skewed now as they were then. Otherwise the "lookback" question of voter recall of their 2011 choice would not result in any significant re-weighting and would not push the Yes vote up. But what the lookback question mainly proves is that he isn't finding enough 2011 SNP voters to ask in the first place. This indicates that the polling sample is insufficiently random.
As to further sub-dividing the SNP part of the sample, any corrections are only valid statistically if the re-weighting is based on strongly dependent variables. One could argue that SNP voting and Yes support are strongly correlated. If not then the SNP has been wasting its time and none of this vote means anything anyway. However, any effort to further subdivide the SNP vote into strong or weak gets into difficulties. What if those Labour supporters who swung to the SNP in 2011 and delivered the landslide were also just thinking people who made up their minds based on the campaign information? Surely those same people would be just as likely not to be mindlessly loyal unionists today, and to vote based on the arguments for and against indy on their merits.
Incidentally, NONE of the polling organisations got even close to the final 2011 SNP regional vote result. The closest of all still put it 5% too low. If you take the most recent six months of polls, knock off YouGov as an outlier, then draw a trendline average through all the other polls, then Yes is less than 5% behind in the poll of polls. Adding the assumtion (a big one, i'll grant you) that based on their 2011 predictions, all the polling organisations consistently underestimated SNP support by at least 5%, then the conclusion can be drawn that Yes are already ahead. A caveat is that a high proportion of those extra SNP voters are not just "devo-maxers" but would also back full indy. If you can think of a lookback question that might drill into that then good luck, the nuance here is as sublime as the referendum debate itself.