Tuesday, February 3, 2015

Why we should maintain a healthy scepticism about tomorrow's Ashcroft constituency polling

11am tomorrow (Wednesday) will herald what in some people's minds is the biggest polling event of the year so far, namely the release of the endlessly-hyped Scottish constituency polls from Ashcroft.  In particular, Mike Smithson over at Stormfront Lite has been investing almost millennial significance in them for the last few months.  I think that's slightly silly - as far as I'm concerned, Scotland-wide polls are more important.  I'll certainly be taking a keen interest in the Ashcroft results, but the following big caveats should be borne in mind...

1) Constituency polling has a very patchy track record.  There's no mystery over why that should be the case - the methodology for national polls can be honed through trial and error, but most of the constituencies we'll be hearing about tomorrow have never been polled in this way before.

2) Ashcroft may weight his results by 2010 recalled vote.  We don't know that for a fact, but it seems conceivable that he'll go down that road in order to keep his methodology consistent with his English constituency polling.  In Scotland-wide polls, it's become accepted good practice to weight by 2011 recalled vote, which we know is much more accurate than 2010 recall.  However, it's not really possible to do that in constituency polls, because nobody knows exactly what the 2011 result was in individual Westminster constituencies (with the exception of the Western Isles and Orkney & Shetland).  That leaves Ashcroft with two basic options - to weight by 2010 recall, or to follow the Ipsos-Mori practice of not weighting by recalled vote at all.  He should do the latter - and if he doesn't, the results will be unreliable, and in all probability will underestimate the SNP.  For the avoidance of doubt, that point will hold true even if the results appear to be very favourable for the SNP.

3) Ashcroft unwisely asks two voting intention questions, and headlines the results of the second question.  In a constituency poll, it's a very good idea to ask respondents to think about how they will vote in the context of the local seat and the local candidates - but that question should be asked upfront.  By asking it second, and immediately after a differently-worded voting intention question, you're inviting people to think they're "supposed" to give two different answers.  If there are several constituencies in tomorrow's polling that produce completely different results on the two questions, then alarm bells should be ringing.  The media will dutifully follow Ashcroft's edict that only the second question matters - but they'll be wrong.  The true position may well be somewhere in between the two results.

I should make clear that I have absolutely no idea at this stage what the polling will show - it may be good, bad, or wonderful for the SNP.  Ashcroft has been having a whale of a time dropping hints, but a lot of them have probably been red herrings.  So I just thought it might be worth airing my concerns well in advance, so there's no question of raising convenient objections to a disappointing result, or falling silent about the flaws in an encouraging poll.

UPDATE : Literally within seconds of me saying that I had no idea what the results were, I was pointed in the direction of what appear to be the results (the Gordon result is bang in line with what Cochrane led us to expect).  The person who sent me them suggested that I shouldn't report the numbers in case they turn out to be wrong.  But suffice to say that Danny Alexander's career prospects are not looking too hot at the moment, even allowing for a huge margin of error.

UPDATE II : As the apparent Ashcroft results have been posted and discussed in the comments section below, I've just gone ahead and published the full numbers in a fresh post HERE.

12 comments:

  1. I fcalled PB Stormfront Lite first. Shame only 3 people use it but a meme's got to start somewhere. Good old rant about Muslims on there today. They never get tired of that one.

    How out of date are the lord ashley polls? Have they not been superseded by notmeguv and populous?

    Are the hacked results from his poll or not. Some rather large swings involved if they are.

    ReplyDelete
  2. "But suffice to say that Danny Alexander's career prospects are not looking too hot at the moment, even allowing for a huge margin of error."

    Also in SHOCK breaking news.. bear shits in woods, pope confirmed catholic. :D

    ROFL

    ReplyDelete
  3. Holy crap - can't believe he's left the results unguarded like that!

    Thats some stunning results right there.

    He should have done Renfrewshire East tho!

    ReplyDelete
    Replies
    1. I'm wondering if there might be a second tranche in future weeks - it's strange that he's done the whole of Glasgow, but nothing in Edinburgh.

      Delete
    2. Boab - "Holy crap - can't believe he's left the results unguarded like that!"

      It's the oldest mistake in the book, leaving "Hidden" directories for easy linking at a later date. If something's on a public website it's by definition not hidden.

      Delete
    3. True. The results in Glasgow are staggering, but obvs thats where some of the biggest swings will be.
      My seat of Dundee West is a mindboggling 27% swing!

      Delete
    4. I was the one who emailed James. I still can't believe that a pollster would leave all the uploads to the site open like that - which is why I said to not to report the figures until Ashcroft publishes them. Although looking through the link - you can find everything that's been uploaded to the site since it was started. That makes me think that they are true...

      Additionally, as Ashcroft ask two voting intention questions - there may be major differences between the results for both questions (some of which may only be noticeable with the data tables).

      Delete
    5. Me: "I still can't believe that a pollster would leave all the uploads to the site open like that"

      or anyone for that matter...

      Delete
    6. Clyde1998 - "I still can't believe that a pollster would leave all the uploads to the site open like that"

      I can. I work in computer security. Human stupidity is the main cause of information security breaches, by far. You'd be amazed at the sheer idiocy I deal with day in, day out.

      Delete
  4. Did he really not do Renfrewshire East or anything in Edinburgh and the Borders or are there more to come?

    Results are... spectacular. Glasgow North East really letting the side down though.

    But Danny... Danny may as well get down the Job Shop. It's not like the Liberals will have the money (or MPs) to even give him a consolation SpAd job.

    ReplyDelete
  5. Letting the side down with a mere 23.5% swing

    Crazy stuff!

    ReplyDelete