You might remember that Calum Findlay revealed in a comment on this blog that Panelbase secretly use the following preamble to the actual referendum question in at least some of their referendum polls, without owning up to it in their results tables -
"As you may know, the Scottish government intends to hold a referendum this year on Scotland becoming a country independent from the rest of the United Kingdom. The question on the ballot is expected to be as below. How would you vote in this referendum?
Should Scotland be an independent country?"
Earlier this afternoon, Alasdair Stirling wrote to Panelbase's Ivor Knox to seek clarification on how often the preamble has been used. To his credit, Mr Knox replied extremely swiftly, indicating that the preamble has been used in "all of our Sunday Times polls", with the obvious exception of the replacement of the words "next year" with "this year". This confirmation is more significant for what it leaves out than for what it says. It presumably implies that the preamble wasn't used for the Wings over Scotland-commissioned Panelbase poll (there have been two Wings polls so far, but only one asked directly for voting intention), or for the SNP-commissioned poll that was conducted in August. That may well indicate that the preamble is artificially boosting the No lead, because of course the SNP poll showed Yes in a one-point lead, well outside Panelbase's normal range. The Wings poll showed an eight-point No lead, which is at the very bottom end of the normal range - it was sandwiched between two polls using the Sunday Times preamble that showed slightly bigger No leads of ten and nine points respectively.
The obvious question that forms in my mind now is - if the preamble is truly only used for the Sunday Times series, is that at the newspaper's suggestion/insistence? Because if so that would obviously cast some doubt on the credibility of the results, given that the Sunday Times has an explicitly anti-independence agenda.
Although it isn't quite as bad as YouGov's notorious Dodgy Preamble which effectively transformed a question about independence into a pejorative one about "leaving the United Kingdom", the Panelbase preamble nevertheless has a number of problems with it that might be pushing some respondents who would otherwise be in the Yes or undecided columns into saying 'No'. Stating that Scotland would be a country independent "from the rest of the United Kingdom" is firmly unionist language, because it conjures up for some people images of breaking links with the monarchy, which is the opposite of the SNP's vision of independence (more's the pity, some of us would say). It is, in any case, negative wording - albeit of a subtler variety than YouGov's. A further subtext of the preamble is that there is still some uncertainty over whether the referendum will even take place, which is clearly no longer the case. And associating the referendum so firmly with the Scottish Government in the opening words may prompt a few respondents to express displeasure with that government rather than with independence itself.
We know what the response of the less thoughtful anti-independence commentators would be to these criticisms - the wording is strictly speaking accurate, so what's the problem? If it really needs to be spelled out, the problem is that accurate wording can very easily be extremely leading, and in either direction. How about...
"Later this year there will be a referendum on whether all of the most important decisions affecting Scotland should in future be made by the Scottish parliament in Edinburgh rather than the UK parliament in London. The question that will be on the ballot is provided below. Do you think you will vote Yes or No?"
Would you be quite so thrilled with that one, chaps?
* * *
UPDATE : I'm going out for the evening, so I'm in a mad rush, but I'll quickly post this email I've just received from Ivor Knox -
"Just seen the report on Scot Goes Pop - I should point out that we used this same wording on the Wings Poll. It originates in earlier polls when "devomax" was still being discussed (so we were describing one scenario within the UK and one separate from the UK) - we then reduced the introduction to its current form once it became a straight Yes/ No to independence.
Hope this helps."
So there's nothing suspicious about the Sunday Times polls generally showing a higher No lead than the Wings poll did, but the question over why the SNP-commissioned poll showed a Yes lead of one point remains. The datasets for that poll clearly seem to suggest that a different preamble was used. As I noted yesterday, commentators were far too quick at the time to leap to the conclusion that the question sequence must have caused a freak result - in retrospect it's just as likely that a different preamble may have produced a better result for Yes.
Of course, the biggest concern of all is over the secrecy surrounding the preamble - most other BPC pollsters (including even YouGov to some extent) have been much more open and transparent about the full wording of the questions they use.
The Sunday Times/Panelbase preamble may not as bad as YouGov's discredited effort, but the effect will be much the same as it's text redolent of the Unionist/anti-independence argument.
ReplyDeleteIt acts, if you will, as a 'dog-whistle' subconsciously reminding respondents of the Unionist/anti-independence argument immediately before they make their decision.
It is the opinion poll equivalent of allowing only Better Together to campaign outside the polling station.
Did RevStu know about the preamble in the Wings poll? Have you asked him?
ReplyDeleteI'm pretty sure Rev Stu wouldn't have allowed the preamble to go ahead, it's pretty disappointing of Panelbase that they have been using a stealth preamble all along.
ReplyDeleteThe reason they have shown the race closer is all to do with methodology in my opinion, I mentioned in another thread my concern that the samples aren't representative and are basically polls of those who voted in 2011 only (so more likely to have an opinion).
On Twitter, Stu confirmed that he knew nothing about such a preamble. Considering how obviously biased it is, I am hardly surprised. However Stu also stated on Twitter that he had contacted Ivor and it was confirmed that they used the preamble without his knowledge.
ReplyDeleteI am both astonished and horrified at the Panelbase decision to use such a slanted preamble. No one could look at it and consider it neutral.
A link to the exchange of Tweets between me and Stu:
ReplyDeletehttps://twitter.com/WingsScotland/status/432991099507187712
I was surprised and a little disappointed when the Wings poll (which immediately followed the SNP one with Panelbase) came back with such a low Yes percentage. I thought the SNP one must have been significantly leading after all, even though I didn't see the order of the questions as a problem - after all, they're the sort of thing voters will be thinking about anyway before they cast their vote.
ReplyDeleteStu is saying on Wings that he doesn't think it's all that leading, but I don't really agree. On the up side, it is now arguable that the SNP poll which put Yes marginally in front is the best estimate of the real state of play.
Rolfe : RevStu is saying that he doesn't see anything sinister in the wording. Judging from Ivor Knox's email, there may well have been no sinister intent, but we have to look at practical effect as well as intent. We know that respondents can be very sensitive to different wordings, and that the closer the question gets to the Cochranesque ideal of "completely separate from the rest of the United Kingdom", the more likely you are to get a response of 'No'. The Panelbase preamble isn't as close to that extreme as the YouGov one, but it's plainly "on the spectrum".
ReplyDeleteWhen YouGov finally introduced a more neutral preamble in September, the No lead immediately slumped by 10%. OK, probably not all of that drop was caused by the new wording, but it's a reasonable assumption that at least some of it was. So at the very least we can't exclude the possibility that the Panelbase preamble is similarly inflating the No lead, and the radically different result of the SNP-commissioned poll is obviously consistent with that suspicion.
RevStu has pointed out that that question wasn't one of the ones we paid for, so I suppose there's no case to complain to Panelbase. However, this has certainly changed the way I view that poll result, which disappointed me at the time as it didn't come close to what the SNP got.
ReplyDeleteI'd love to see an anonymous internet poll from someone like Panelbase with a completely neutral preamble. I think we might be close to parity by now.
Another confounder is the weighting by previous vote - even for Holyrood. The 2011 turnout wasn't much in excess of 50%. The people who didn't vote aren't really showing up in these statistics. Do we really think only 50% are going to turn out in September? Do we think the non-voters in previous elections are going to break for No in the referendum? I don't.
ReplyDeleteRolfe, only 14% of the WOS Panelbase sample didn't vote in 2011 - the turnout was only 50% at the election. YouGov and ICM have around 20% non voters, which is better but I'm still doubtful about representative the online pollsters can be. That methodology would be fine if they were polling for an election, but as you've said, many people who don't normally vote will in the referendum.
ReplyDeleteThe main explanation that's been put forward for the SNP's "Yes 1 point ahead" poll is that it contained some leading questions immediately beforehand.
ReplyDeleteSpecifically, it asked whether "Scotland could be a successful independent country" (responses to this are always very positive).
Then it asked whether respondents trusted the Scottish government more than Westminster (most people say they trust the Scottish government more when this question is asked).
It was only after those two questions that they asked whether people supported independence. It's well established that the questions you ask immediately beforehand in a poll can lead people in a certain direction - given how out of synch that poll is with all the others that would appear to be a reasonable explanation for it.
None of which is to say that all of the other polls are perfect, but this is why it's useful to use averages from different polls rather than relying on any one in particular.
"It's well established that the questions you ask immediately beforehand in a poll can lead people in a certain direction"
ReplyDeleteThe operative word there being 'can'. It's equally well established that the wording of a question can lead people in a certain direction. The poll showing Yes in the lead used a different and much more neutral preamble, so as things stand we simply have no way of knowing whether the question sequence or the neutrality of the wording was responsible for producing such a radically different result. It may, of course, have been a combination of both factors.
@James Of course the wording of the question/preamble is also important, but what we're talking about here is one poll that showed the Yes vote a significant margin ahead of any other poll that's been done on the subject in the past two years. Operating from the standpoint that it did something to raise the Yes vote in comparison to the rest of the polls is pretty reasonable (we have a pretty solid hypothesis given that no other poll I know of asked those questions beforehand, whereas plenty of other polls have used neutral wording in the preamble).
ReplyDeleteThe same thing can also be said of the polls that showed extremely large levels of support for No, which is why an average from polls gives a better picture.
"Operating from the standpoint that it did something to raise the Yes vote in comparison to the rest of the polls is pretty reasonable"
ReplyDeletePrecisely. And what sticks out like a sore thumb about that poll is that we now know that it was the ONLY Panelbase poll not to use the subtly biased preamble quoted at the top of this post. Yes, it was also unusual in that it asked two questions (which were not leading, by the way) before the main refetendum question. So we have two equally plausible explanations for the radically different result, and literally no way of knowing which is the most likely.
On your wider point, trying to compare that poll with non-Panelbase polls is like comparing apples with oranges.
"On your wider point, trying to compare that poll with non-Panelbase polls is like comparing apples with oranges."
ReplyDeleteIt's a standard opinion polling technique that's been used successfully in countless other contexts. The point being that every poll/polling agency has some potential flaw in its methodology and by pooling them you can eliminate most of the noise and get a more accurate picture of voters' actual opinions(e.g. using Bayesian methods similar to Rob Ford and other academics: http://pollingobservatory.wordpress.com/methodology/).
So let's be clear about what we're actually implying here. This isn't about the natural uncertainty over how one particular bit of wording or question sequence can skew the results; what we're saying is that this one isolated Panelbase survey has the most neutral wording/question sequence of all the professional polls published on the subject in the past two years. That's the only reason we could have not to use an average - i.e. if all of the other polls are systematically flawed or subject to bias in some way.
Incidentally, I completely agree with RevStu that the preamble isn't particularly leading in any case. It's only marginally different from the preamble used in the SNP poll. Instead of saying independent from the UK it says an "independent Scotland" - that might on some subtle level have an impact (and potentially a misleading impact at that), but it's a bit far fetched to expect that to cause the kind of radical swing we're discussing here. We'd certainly need far more evidence to go on before arriving at that conclusion and effectively writing off methods like the one I've linked to above.
Don't get me wrong, it would be nice if you were correct, but from a political science perspective I think we're a bit out on a limb here.
"The point being that every poll/polling agency has some potential flaw in its methodology and by pooling them you can eliminate most of the noise and get a more accurate picture of voters' actual opinions"
ReplyDeleteThat seems to me like a utopian vision of what it is possible do with statistics. The reality is that there are countless examples around the world of the "extreme outlier" poll turning out to be the correct one, and it usually happens when most pollsters share certain methodological mistakes in common, and are reinforced in their false belief that they are getting it right by the results produced by their competitors. In a UK context, the most obvious examples are 1970, where just one eve-of-election poll had the Tories ahead, and 1992, where all pollsters underestimated the Tories' true position to such an extent that even the most extreme outlier during the entire campaign failed to predict how convincing the Tory win would be. Lesson : pollsters should be going back to basics and concentrating on getting their own methodology right, rather than slipping into the kind of group-think you are suggesting of assuming that they must be right if they're producing numbers that are somewhere close to the average.
You're talking as if the preamble issue is the only criticism anyone is making of the other pollsters. It isn't. For example, it was just two days ago that Scottish Skier pointed out that YouGov have a disproportionately high number of people in their sample who were born outside Scotland, and have a suspiciously low number of people who choose "Scottish" as one of their national identities. Similar criticism has been levelled at Ipsos-Mori (who incidentally also appear to use a 'stealth preamble', so we have no way of judging how neutral or leading it may be). Survation's methodology in their one poll to date was so plainly wrong that even Professor Curtice didn't mince his words - they were clearly understating Yes and overstating No.
I think you (and RevStu if he takes the same view) are absolutely, fundamentally wrong in believing that this Panelbase preamble is innocuous. We have considerable evidence that the wording most likely to lead respondents towards No is "completely separate from the rest of the United Kingdom". As I've stated several times now, this preamble isn't quite as extreme as that, and you're quite correct that the word 'independent' is neutral in a way that 'separate' is not. But the words "from the rest of the United Kingdom" are present, which means that this preamble is 'on the spectrum' of bias. Those words are entirely superfluous in explanatory terms, so what you have to ask yourself is this - what are they actually doing there, and what effect are they having? Unless you have hard evidence that they are having no effect, it's naive and complacent to assume that must be the case.
"That's the only reason we could have not to use an average"
I assume from those words that this must be your first visit here, because I've been running a Poll of Polls based on three different averages of BPC pollsters for several months now - and to the best of my knowledge I'm the only person doing anything like that in such a structured way at this stage. But I pointed out from the outset that the best claim that can be made for an average is merely that it is likely to be less inaccurate than any other system that might be devised, as opposed to there being any firm basis for thinking that it is necessarily going to be particularly accurate. To go back to the 1970 example, an average of the polls would have shown a picture much further from the truth than the one outlying poll. The BBC Poll of Polls in 1992 showed a Labour lead of 1% on the eve of polling day. Actual result? Tory lead of 8%. Again, the most extreme outlying poll was closer to the truth.
I've reposted the exchange with Bill in a fresh post, so if anyone has any follow-up comments it would probably be best to leave them there.
ReplyDelete