You might remember that Calum Findlay revealed in a comment on this blog that Panelbase secretly use the following preamble to the actual referendum question in at least some of their referendum polls, without owning up to it in their results tables -
"As you may know, the Scottish government intends to hold a referendum this year on Scotland becoming a country independent from the rest of the United Kingdom. The question on the ballot is expected to be as below. How would you vote in this referendum?
Should Scotland be an independent country?"
Earlier this afternoon, Alasdair Stirling wrote to Panelbase's Ivor Knox to seek clarification on how often the preamble has been used. To his credit, Mr Knox replied extremely swiftly, indicating that the preamble has been used in "all of our Sunday Times polls", with the obvious exception of the replacement of the words "next year" with "this year". This confirmation is more significant for what it leaves out than for what it says. It presumably implies that the preamble wasn't used for the Wings over Scotland-commissioned Panelbase poll (there have been two Wings polls so far, but only one asked directly for voting intention), or for the SNP-commissioned poll that was conducted in August. That may well indicate that the preamble is artificially boosting the No lead, because of course the SNP poll showed Yes in a one-point lead, well outside Panelbase's normal range. The Wings poll showed an eight-point No lead, which is at the very bottom end of the normal range - it was sandwiched between two polls using the Sunday Times preamble that showed slightly bigger No leads of ten and nine points respectively.
The obvious question that forms in my mind now is - if the preamble is truly only used for the Sunday Times series, is that at the newspaper's suggestion/insistence? Because if so that would obviously cast some doubt on the credibility of the results, given that the Sunday Times has an explicitly anti-independence agenda.
Although it isn't quite as bad as YouGov's notorious Dodgy Preamble which effectively transformed a question about independence into a pejorative one about "leaving the United Kingdom", the Panelbase preamble nevertheless has a number of problems with it that might be pushing some respondents who would otherwise be in the Yes or undecided columns into saying 'No'. Stating that Scotland would be a country independent "from the rest of the United Kingdom" is firmly unionist language, because it conjures up for some people images of breaking links with the monarchy, which is the opposite of the SNP's vision of independence (more's the pity, some of us would say). It is, in any case, negative wording - albeit of a subtler variety than YouGov's. A further subtext of the preamble is that there is still some uncertainty over whether the referendum will even take place, which is clearly no longer the case. And associating the referendum so firmly with the Scottish Government in the opening words may prompt a few respondents to express displeasure with that government rather than with independence itself.
We know what the response of the less thoughtful anti-independence commentators would be to these criticisms - the wording is strictly speaking accurate, so what's the problem? If it really needs to be spelled out, the problem is that accurate wording can very easily be extremely leading, and in either direction. How about...
"Later this year there will be a referendum on whether all of the most important decisions affecting Scotland should in future be made by the Scottish parliament in Edinburgh rather than the UK parliament in London. The question that will be on the ballot is provided below. Do you think you will vote Yes or No?"
Would you be quite so thrilled with that one, chaps?
* * *
UPDATE : I'm going out for the evening, so I'm in a mad rush, but I'll quickly post this email I've just received from Ivor Knox -
"Just seen the report on Scot Goes Pop - I should point out that we used this same wording on the Wings Poll. It originates in earlier polls when "devomax" was still being discussed (so we were describing one scenario within the UK and one separate from the UK) - we then reduced the introduction to its current form once it became a straight Yes/ No to independence.
Hope this helps."
So there's nothing suspicious about the Sunday Times polls generally showing a higher No lead than the Wings poll did, but the question over why the SNP-commissioned poll showed a Yes lead of one point remains. The datasets for that poll clearly seem to suggest that a different preamble was used. As I noted yesterday, commentators were far too quick at the time to leap to the conclusion that the question sequence must have caused a freak result - in retrospect it's just as likely that a different preamble may have produced a better result for Yes.
Of course, the biggest concern of all is over the secrecy surrounding the preamble - most other BPC pollsters (including even YouGov to some extent) have been much more open and transparent about the full wording of the questions they use.