"On a scale of confusion from 0 to 10..."

Regular readers might remember that back in the spring of this year, when Progress Scotland published its first poll, I pointed out that the Sun newspaper had misreported it (probably at the prompting of Scotland in Union) as showing that support for independence had "dropped below 40%".  It was perfectly true that support appeared to be lower than 40%, but the inaccurate word was "dropped".  In fact, the question format was completely different from standard independence polls, meaning it was impossible to make a comparison with those polls and conclude that the Yes vote had either risen or fallen.  There was no way of knowing whether earlier polls using the same format would have shown greater or lesser support for independence, although it did seem pretty likely that there was something about the question that was producing less favourable numbers than a Yes/No question would.

Basically respondents were asked to express their degree of support for independence on a scale of 0 to 10, with 0 indicating total support and 10 indicating total opposition.  I speculated at the time as to why that might have produced an artificially low result for independence, but now we have a stronger clue.  The pro-independence Twitter legend David Halliday contacted me three weeks ago to say that he'd just taken part in a Survation poll using exactly that question format - and that he'd accidentally indicated total opposition to independence.

"Just had a really interesting - if embarrassing - experience taking a Survation poll. It started out about me - where I lived (which region in Scotland - clearly Scotland only), age, children, income - then on to how I voted in 2017, 2016 and 2014. Then - this is the interesting and embarrassing bit - it asked a question about independence on a scale of 0 or 1 to 10. I went straight to 10 ("Totally in favour of") and clicked next and only then realised that 10 was "Totally in favour of staying within the UK" (or similar) while 0 or 1 was "Totally in favour of independence". I tried to go back but couldn't so stopped the poll, in the hope my vote won't be counted. A real life example of how the wording of the question in a different way (and one that confusingly harked back to the 1 to 10 questions Yes canvassers asked in 2014 where the 10 was dead in favour of independence) can skew the result. I'm wondering if it was deliberate."

It's obviously unlikely that any poll commissioned by Angus Robertson would have tried to achieve that effect deliberately, but it does illustrate why any numbers produced by this question format should be taken with a pinch of salt.  If someone as intelligent and politically-engaged as David was capable of misreading the question and saying the opposite of what he intended to say, it doesn't take much of a leap of imagination to suppose that plenty of other respondents may have done exactly the same thing.  Even if you hadn't encountered the 1 to 10 questions asked by Yes canvassers in 2014, it's entirely natural to assume that the highest number would indicate the maximum support for the proposition you are being asked about.

Here is a screenshot that David took of the question when he managed to revisit the poll later - would you have been confused?  Particularly if you weren't looking too carefully?

Based on other questions that were asked, David is very confident that the poll he was taking was the latest one for Progress Scotland, and yet oddly the datasets for that poll suggest that the question was asked the other way around, with 10 indicating total support for independence and 0 indicating total opposition.  So what's going on?  Are the datasets wrong?  Were two different halves of the sample asked the question in different ways?  No idea.  For what it's worth, though, this poll is slightly better for Yes than the one in the spring, with exactly 40% of respondents putting themselves on the independence-friendly end of the spectrum, and another 6% choosing the neutral option of 5.