Wednesday, December 11, 2024

An unweighted Yes vote of 60.4% in the Norstat poll became 53.8% in the weighted results

Similar to that sensation of 'feeling your ears burning', I can sometimes sense when Professor John Robertson of Global Ferry News fame is about to write a snarky blogpost about Scot Goes Pop, because it's often presaged by him attempting to leave a comment here.  There was a comment yesterday or the day before with all the usual hallmarks, so I took a look at his blog, and this time there was no post about SGP, but what I did find was a post with the following headline: "Did the Sunday Times hide around 60% support for independence by dramatically reducing the number of 2014 Yes voters in their poll sample from 387 down to only 278?"

That's what John Rentoul would call a QTWTAIN (Question To Which The Answer Is No).  Of course the Sunday Times did no such thing - they have no role in determining the weightings used in any poll, they would never think of making such a request and Norstat would refuse such a request if it was made.  That said, I've looked at the data tables and it's true that the difference between the unweighted and the weighted numbers is pretty extreme.  The Yes vote in the unweighted numbers is 60.4%, whereas in the weighted numbers it's 53.8%.  OK, even normal demographic weightings can often bring down the Yes vote, because there might be too many young people in a sample, or too many SNP voters, or whatever.  But I very much doubt if that sort of thing would have had quite such a dramatic effect - the major explanation in this case is almost certainly direct weighting by each respondent's recollection of how they voted in the 2014 referendum, which of course took place more than a decade ago, thus opening up a risk of significant levels of false recall.  That's one of the reasons Ipsos have cited for not weighting by recalled 2014 vote, incidentally.

In the overall Norstat sample, before the likelihood to vote filter is applied, 43.9% of respondents claim to recall voting Yes in 2014, and only 30.0% claim to recall voting No.  A drastic adjustment has been made to bring those numbers into line with the actual 2014 result.  In fairness, there was initially a very good reason for introducing 2014 weighting, because polling companies had systemically overestimated the Yes vote by a small amount in the 2014 campaign.  But after more than a decade that adjustment is getting harder and harder to justify, and it's impossible to rule out the possibility that it may be artificially skewing poll results towards No and giving us a totally false impression of the state of play.  It must be very unusual to weight poll results by an electoral event that took place more than ten years ago - I'm struggling to think of any other examples of that happening, even internationally.  Somewhere in the deepest recesses of their minds, the heads of polling firms must be gearing up towards a review of this problem, sooner or later.  

*  *  *

We should now think about calling a national holiday, because that rarest of rare things has just happened - Chris McEleny has actually responded to an email.  Apparently the clerk of the committee will be in touch in due course about the arrangements for my appeal against expulsion from the Alba Party.  I'll be interested to see what "in due course" actually means, because from my recollection of the party constitution (which I can no longer read because I'm barred from the party website), the appeal is supposed to be heard within an extremely tight timetable.

*  *  *

If you find Scot Goes Pop useful and would like to help it to continue, donations by card payment are welcome HERE, or alternatively donations can be made direct by PayPal.  My PayPal email address is:  jkellysta@yahoo.co.uk

26 comments:

  1. ALBA Party Constitution if you want a copy of it: https://www.scribd.com/document/803561668/ALBA-Party-Constitution

    ReplyDelete
  2. The sun will expand to become a red giant, swallowing up planet earth before running out of hydrogen fuel and collapsing in on itself....in due course.

    ReplyDelete
  3. We shall soon have a better sense of the real polling picture. Yes now ahead, by my reckoning, but … maybe not. And even if so, by what margin?

    ReplyDelete
  4. I cant begin to imagine false recall of having voted yes in 2014.

    ReplyDelete
  5. James - There is an Indy Supporting poll analyst on twitter who has been raising this points for months, if not years. He sporadically posts Indy polls without using the 2014 weighting and virtually all of them gave Yes in a substantial lead.
    Are you aware of him? ( his twitter name escapes me for the moment )

    ReplyDelete
    Replies
    1. Are you talking about Independent Voices? I've seen him do it once or twice, but I didn't doublecheck his figures, so I don't know how accurate his claims are. (I'm not saying they're not accurate, I just don't know one way or the other.)

      Delete
    2. Maybe.
      Will check who I follow/my followers on there.
      He is in there somewhere.

      Delete
    3. It was indeed 'Independent Voices'.
      He left twitter and is now on Bluesky.
      Perhaps worth a look, if you have time - as you both seem to have legit questions on Indy-Poll weighting.

      Delete
  6. Lol.
    But.......what if the weighting was suspect/wrong?
    Most of those polls were within 'margin of error' territory and would quite easily swing to Yes if the weighting used was No-Biased.
    I think James, Robertson a others might be correct in doubting the veracity in 2024/5 of a weighting-system so heavily dependent on 2014 and which may well not take into account the substantial demographic changes in the last decade.
    Certainly bears further examination.

    ReplyDelete
  7. I think the Norstst (Panelbase) panel is full of die hard Indy supporters. If you remember lots of Yes supporters joined Panelbase prior to the 2014 referendum, there was a bit of a campaign to get people to sign up. I reckon they are mostly still there

    The Sunday Times using Nortstat rather than YouGov or Survation is probably to give the SNP false confidence ahead of a hammering in 2026

    ReplyDelete
    Replies
    1. The Sunday Times have used Norstat (or their predecessor Panelbase) for years and years and years - it's not a new thing at all.

      Delete
    2. Have to agree with you Denise.
      I think there’s little doubt the Norstat poll is a bit of an outlier, hopefully not but it looks that way.
      The next polls from the likes of YouGov and Survation will be eagerly awaited am sure.

      Delete
    3. That's interesting, KC, but most of us will be showing keener interest in what the more credible pollsters like OpinoSpa and Favesurv show. Fascinating that Norstat has finally come into line with them by showing a Yes lead.

      Delete
  8. Fascinating that even the mere notion that the weighting system used by most pollsters in relation to Indy Polls 'might' be erroneously skewed towards No, can cause some folk to break out in hives.
    And not just unionists either...........

    ReplyDelete
    Replies
    1. Time will certainly tell whether the Norstat poll was an outlier.
      It was certainly a bit of much needed good news but I think we should exercise caution.

      Delete
  9. I suppose the suggestion here is that the reason for 'false recall' is guilt or wishful thinking - that is, that people aren't *really* forgetting that they voted No, it's just that they regret their vote and want to pretend they voted Yes. Otherwise, why would false recall not affect both 2014 Yes and No voters equally?

    Has there been detailed research on this and other potential reasons for false recall which might not risk skewing present-day results by quite so much?

    ReplyDelete
  10. There is of course one rather large elephant somewhere in the room.
    What if……… the given result from 2014 is not what the actual result was!
    I saw plenty evidence on the day, following the count on TV and in front of my face before I left my local count that the result as presented was definitely pauchled.
    I’m not saying we won outright , but I’d bet my life there was no 55/45 split in that vote
    Hence recall showing a higher Yes vote is a perfectly normal thing.
    What we need is for current real figures to be in the public domain
    Constantly being 2% or whatever behind is just part of the Brit establishment’s agenda

    ReplyDelete
    Replies
    1. I suggest: go lie down in a dark room and try to pull yourself back together, pal

      Delete
    2. Truly desperate stuff @9:43.

      Delete
    3. Don't forget the big burly men at the count. NOBODY should forget them.

      Delete
  11. Quite incredible bullshit from some on here, suggesting the polls are wrong and there is a majority in favour of independence!!!
    Dear oh dear, Nats becoming ever more desperate, unable to accept their “dream” has died.

    ReplyDelete
    Replies
    1. I agree with you, the Brit Nat activists are becoming increasingly desperate. Perhaps it's because in the last 12 months keeping in the undecideds, only 1 single poll had NO at 50% or more - and that was the same company and client as this one. Norstat for The Sunday Times (-9–12 Apr 2024).

      And it's the end of the Union as we know it. Union no more!

      Delete
  12. McEleny is big fat bassa

    ReplyDelete
  13. @9.54, 11.09, 11.33 and 11.39pm
    To paraphrase
    “I love the smell of desperation in the morning
    It smells like victory “

    ReplyDelete
    Replies
    1. You're Ruth Davidson and I claim my £5.

      Delete