Monday, October 21, 2019

"On a scale of confusion from 0 to 10..."

Regular readers might remember that back in the spring of this year, when Progress Scotland published its first poll, I pointed out that the Sun newspaper had misreported it (probably at the prompting of Scotland in Union) as showing that support for independence had "dropped below 40%".  It was perfectly true that support appeared to be lower than 40%, but the inaccurate word was "dropped".  In fact, the question format was completely different from standard independence polls, meaning it was impossible to make a comparison with those polls and conclude that the Yes vote had either risen or fallen.  There was no way of knowing whether earlier polls using the same format would have shown greater or lesser support for independence, although it did seem pretty likely that there was something about the question that was producing less favourable numbers than a Yes/No question would.

Basically respondents were asked to express their degree of support for independence on a scale of 0 to 10, with 0 indicating total support and 10 indicating total opposition.  I speculated at the time as to why that might have produced an artificially low result for independence, but now we have a stronger clue.  The pro-independence Twitter legend David Halliday contacted me three weeks ago to say that he'd just taken part in a Survation poll using exactly that question format - and that he'd accidentally indicated total opposition to independence.

"Just had a really interesting - if embarrassing - experience taking a Survation poll. It started out about me - where I lived (which region in Scotland - clearly Scotland only), age, children, income - then on to how I voted in 2017, 2016 and 2014. Then - this is the interesting and embarrassing bit - it asked a question about independence on a scale of 0 or 1 to 10. I went straight to 10 ("Totally in favour of") and clicked next and only then realised that 10 was "Totally in favour of staying within the UK" (or similar) while 0 or 1 was "Totally in favour of independence". I tried to go back but couldn't so stopped the poll, in the hope my vote won't be counted. A real life example of how the wording of the question in a different way (and one that confusingly harked back to the 1 to 10 questions Yes canvassers asked in 2014 where the 10 was dead in favour of independence) can skew the result. I'm wondering if it was deliberate."

It's obviously unlikely that any poll commissioned by Angus Robertson would have tried to achieve that effect deliberately, but it does illustrate why any numbers produced by this question format should be taken with a pinch of salt.  If someone as intelligent and politically-engaged as David was capable of misreading the question and saying the opposite of what he intended to say, it doesn't take much of a leap of imagination to suppose that plenty of other respondents may have done exactly the same thing.  Even if you hadn't encountered the 1 to 10 questions asked by Yes canvassers in 2014, it's entirely natural to assume that the highest number would indicate the maximum support for the proposition you are being asked about.

Here is a screenshot that David took of the question when he managed to revisit the poll later - would you have been confused?  Particularly if you weren't looking too carefully?


Based on other questions that were asked, David is very confident that the poll he was taking was the latest one for Progress Scotland, and yet oddly the datasets for that poll suggest that the question was asked the other way around, with 10 indicating total support for independence and 0 indicating total opposition.  So what's going on?  Are the datasets wrong?  Were two different halves of the sample asked the question in different ways?  No idea.  For what it's worth, though, this poll is slightly better for Yes than the one in the spring, with exactly 40% of respondents putting themselves on the independence-friendly end of the spectrum, and another 6% choosing the neutral option of 5.

35 comments:

  1. It'd be nice to see an article on this:

    https://lordashcroftpolls.com/2019/10/england-and-the-union/

    It's very meaty!

    ReplyDelete
  2. All polls about independence should have one question so as not to muddy the answer. For independence or against independence.

    ReplyDelete
    Replies
    1. In addition to Yes Scotland, in my day the SNP used to ask canvassees for a figure of 1-10 to indicate level of support for independence. Maybe they still do. No idea why Angus Robertson approved that scale being reversed, though. That poll in general was a pretty horrible own goal from him.

      Delete
  3. It's at 50% folk say, that's not enough folk say, well yes that is enough because that doesn't mean 50% are against Independence, never in the history of voting ever even in Australia where it's compulsory do 100% of people vote

    Around 15% of people everywhere in the world in every country don't vote and never vote because they either feel they don't care or their votes don't count or some other reason beyond my comprehension but they just don't vote

    So on that basis alone 50% for Independence when you take out the don't knows and the never voters ends up a very healthy plus

    ReplyDelete
    Replies
    1. Headline figures from independence polls already exclude Don't Knows and Won't Votes. So when it's 50% Yes, that means it's also 50% No.

      Delete
    2. In reality young James the percentage vote does not mean anything to the Scottish Nat sis and anti brexiteers. They do not accept losing. In reality you are fascists. Boris should impose the leave vote and call in the military against the fascists if required. We do not need Knickerless goosesteppers in the UK.

      Delete
  4. I'm 50% confused by all this.

    ReplyDelete
    Replies
    1. Most alleged Scottish Nat sis are confused because they did not originate in Scotland.

      Delete
    2. Nobody "originates in Scotland", you wazzick. We all originated in Africa. Most of the offspring of those people developed sentience, but there MUST have been some branches which were less successful in that regard, because you are evidence of the existence of one of those less successful branches......

      Delete
    3. The Scots came fae Ireland. The English fae Germany.

      Delete
    4. The Picts are from Pictland, Stu Campbell is from Mars...

      Delete
    5. Not sure, but I think it might be Natalie's sister?

      Delete
    6. I think he gets confused easily, a bit like unionists trying to understand economics.

      Also have you heard about the poll in the scotsman.

      Delete
    7. It's exactly the same poll mentioned in this blogpost.

      Delete
  5. Don't you know you were originally black GWC, we all were but after a very long time of being in northern hemispheres with less sunlight we all got lighter in colour and because of the cold and higher oxygen levels our noses grew thinner until we looked like we are now

    I guess that must be upsetting for the likes of you

    ReplyDelete
  6. I'm 10/10 for indy. It's my No. 1 priority.

    What a dumb ass question.

    Any question that makes the participant have to pause and think to ensure they answer it the correct way is an automatic fail in polling.

    The only way such scaling questions will have some accuracy is when you leave out the numbers.

    ReplyDelete
  7. What an absolute battering Johnson's English EVEL MPs are giving celtic nation unionists.

    When he knocked out Ruth Davidson I was impressed, but forcing the border down the Irish sea + gay marriage + legalised abortion on the N. Irish unionists all in the space of a few days is truly something.

    They must love the union right now... Feel real bonds with the English.

    ReplyDelete
    Replies
    1. Is Colonel Windbag still an MSP or has she done a Dugdale?

      Delete
  8. The question in the poll is perfectly clear to people who can read, James.

    ReplyDelete
    Replies
    1. You seem to have catastrophically misunderstood the point of the blogpost, Anon. The question is not whether in your pompous opinion people *should* have misunderstood the question, but whether or not they *did*. And as there is evidence they *did*, your observation is - alas - gloriously irrelevant.

      Delete
    2. If you need a reasonable level of literacy / numeracy, and have to take a moment or two to understand the question / how to plae your answer, it's not going to be reliable in polling.

      If the participant has to even think for a second about how to correctly put their answer, you have a problem.

      1 person in 100 gets it wrong and you are 1% out.

      90 get it right but 10 tick the wrong box when flying through the poll for their yougov £ credits on the coffee break at work and you have a 10% lead for X over Y where none exists.

      Delete
    3. ~20% of adults have varying problems with literacy.

      https://literacytrust.org.uk/parents-and-families/adult-literacy/

      It's a serious problem for polling.


      Delete
    4. I'll be here in a tick to do something about literacy!

      But do you think I should rebrand as Leonard Coherent?

      https://twitter.com/LabourRichard/status/1179703944953630721

      Delete
  9. The pre-amble to any indy poll should aiblins be.. Do you really think the sleekit law-defying honourable members of the UK Govt. who are destroying England's future in the name of democracy should have the power to do the same to Scotland.
    Given the chance would you vote for Scotland to become an independent country. YES or no.

    ReplyDelete
  10. There should be portrait of President Jamez on each ballot paper. Then we end this comunista democracia nonsense.

    Viva El Presidente!

    ReplyDelete
  11. I wonder if there might be a blogpost on Quebec and its neverendum coming up shortly. (Is there a Quebecois version of Wings?)

    ReplyDelete
    Replies
    1. Or the British referendum on Europe. What happened to the matter being settled in 1975! Campaigning for an exit began again the day after!

      GWC refused to accept the result from day 1.

      Delete
  12. President Jamez is at one with his country.

    He rambles among mountain lochs.

    His favourite food is stovies and his favourite flower is the bluebell.

    ReplyDelete
  13. Another 35% SNP subsample with YouGov. They've went from being the most bullish to being amongst the weakest for the SNP

    ReplyDelete
    Replies
    1. Seats:
      42(+7) SNP
      11(-2) Con
      5(+1) Lib
      1(-6) Lab

      71% of Seats SNP. An epic landslide for them if it came to pass.

      Delete
  14. Liberal Democrats will go down not up and likely only be left with 1 MP Orkney as usual
    Swinson is predicted to lose badly in East Dunbartonshire, they're probably looking for a shoe in seat for her in England as we speak
    Alister Jack is also predicted to be wiped out along with around 8 more Tories even Labour's Tory boy Ian Murray's coat is on a shoogly peg

    The SNP look to be on course for over 55 seats and possibly even more than the 56 sensation landslide they had before

    ReplyDelete
    Replies
    1. Meant to ask - HoC was supposed to be down to 600 seats and Scotland to 50. Think that was about a year ago. Wha'appen?

      Delete
    2. The Liberal Democrats will not end up with one seat. Jo Swinson will not lose in East Dunbartonshire. It's counter-productive to say she will, because it just allows the Lib Dems to win the expectation game. The SNP are on course to make significant gains, but from the Tories and Labour, not from the Lib Dems.

      Delete