Friday, October 3, 2014

Pondering a Panelbase puzzler

As Scottish Skier has been repeatedly cautioning us, it's difficult to have a lot of confidence in the weighting procedures Panelbase used for the full-scale Scottish poll they produced yesterday.  For the avoidance of doubt, that's good news for the SNP, because by and large the weighting didn't work in their favour.  The 282 respondents who recalled voting SNP in the 2010 Westminster general election were downweighted massively to count as just 162 people, and yet the SNP still ended up leading Labour by 34% to 32% in the final weighted results.  It's impossible to calculate what the lead would have been if a more sensible weighting system had been used, but it's safe to assume it would have been at least somewhat bigger.

As regular readers know, the problem with weighting by recalled vote from 2010 is that there's quite strong evidence that people who voted Labour or Lib Dem in 2010 and then switched to the SNP in 2011 get mixed up between the two votes, and understandably the more recent one is fresher in their memory.  Over the course of the long referendum campaign, the discredited practice of 2010 weighting was gradually consigned to the scrapheap, and indeed when Survation burst on to the scene at the start of this year with a poll that used 2010 weighting, they were castigated for it by Professor Curtice and changed their procedure in their very next poll.  So it's a bit of a mystery why Panelbase have suddenly decided to turn the clock back to the bad old days.  Even stranger is the fact that they've done it while maintaining their previous weighting procedure for Scottish Parliament voting intention, which means that this poll is almost unique in having Holyrood and Westminster numbers that are not directly comparable, ie. we have no idea what the difference would be between the SNP's vote for each parliament if the same weighting had been used consistently.

The only thin logic I can see for this is that the level of turnout is different for each type of election, and Panelbase might think that this somehow makes 2010 recall most appropriate for Westminster elections, 2011 recall most appropriate for Holyrood elections, and so on.  But if that is the reasoning, why do they taint their Holyrood weighting by adding on European Parliament recalled vote weighting as well?  Presumably it's because they intuitively feel that 2011 recall is leading to the SNP being weighted up too much, and there needs to be some counterbalance.  Why, then, do they not feel any need for a similar counterbalance to the obviously silly amount by which the SNP have been downweighted in Westminster voting intention?  It just doesn't make any sense.

It's important to note, though, that 2010 weighting doesn't in itself explain away the fact that the SNP's Westminster lead in the Panelbase poll is lower than it has been in most recent Scottish subsamples of GB-wide polls, because those polls are also generally weighted either by 2010 vote or by something very similar (albeit the weighting occurs at a GB-wide rather than Scottish level).

Incidentally, the long sequence of subsamples putting the SNP in the lead has now been broken - a Populus poll conducted yesterday and the day before puts the party on 35%, three points behind Labour.  But today's YouGov subsample, which has identical fieldwork dates to Populus, continues to show the SNP in the lead, albeit by a relatively narrow 39% to 35% margin.

*  *  *


At this point my plan of giving much greater weighting to full-scale Scottish polls in the Poll of Polls kicks in, which inevitably means that the SNP's lead is sharply down, because the Panelbase poll constitutes more than half of the entire sample.  However at least we still have a continuing SNP lead for Westminster now that the sample is more credible in nature.  The following numbers take nine polls into account - the full-scale poll from Panelbase, plus four YouGov subsamples, two Populus subsamples, one ComRes subsample and one Ashcroft subsample.  Irritatingly, I can't provide a figure for the Greens for the time being, because Panelbase have lumped them in with "others" for Westminster voting intention, even in the datasets.

Scottish voting intentions for the May 2015 UK general election :

SNP 35.6% (-2.8)
Labour 31.3% (+3.2)
Conservatives 17.2% (+0.1)
Liberal Democrats 5.8% (-1.3)
UKIP 5.4% (+0.5)

(The Poll of Polls uses the Scottish subsamples from all GB-wide polls that have been conducted entirely within the last seven days and for which datasets have been provided, and also all full-scale Scottish polls that have been conducted at least partly within the last seven days. Full-scale polls are given ten times the weighting of subsamples.)

Last but not least, many thanks to Sandy and Jock for putting together some Poll of Polls graphs yesterday - links to them can be found on the last thread.  I didn't get a chance to post them properly because I was out all day, and of course they've now been superceded by the new update.


  1. The unweighted base that said they voted like this in 2011 V#1:

    42(-3)% SNP
    31(-1)% Lab
    14(nc)% Con
    8(nc)% Lib
    4(+4)% Green
    2% Other

    Said roughly this for 2015:

    17.1% Con
    25.6% Lab
    3.8% Lib
    42.9% SNP
    6.7% UKIP
    3.8% Other

  2. "Presumably it's because they intuitively feel that 2011 recall is leading to the SNP being weighted up too much, and there needs to be some counterbalance."

    "intuitively"? Is that the same as fixing the results to suit your prejudices?

  3. I really feel that far too much emphasis is being put on possible under valuing of SNP votes. I followed this blog religiously before the referendum and got my hopes up because of the constant chat of underestimating the Yes vote. The eve of referendum polls mostly were at 48% yes so given all the chat about the Kelner effect etc I thought we might win. Yet the swing went the other way. Maybe I'm still hurting too much about the loss but my thoughts right now are that we should probably assume the Yes and SNP vote are actually overestimated. Then we'll know how much work we all need to do to secure a future Yes vote.

  4. I agree with Alan on this one, it works in our favour to be overtly conservative when it comes to our predictions.

  5. @Alan

    As James did in iref polls and is doing with Westminster VI.

    James is a total conservative for polls and I'm happy with that.

  6. Alan, just a few weeks before polling day, YouGov (complete with Kellner Correction) had Yes on 39%, Ipsos-Mori had Yes on 40% and TNS had Yes on 42%. The eventual Yes vote was 45%. There might have been a genuine swing on that scale, but given that other pollsters were showing Yes at a much higher level at a much earlier stage, we have to consider the possibility that the three No-friendly firms were indeed underestimating Yes all along. The idea that the final result somehow proves that Yes wasn't being underestimated is quite wrong (except in the limited sense that the post-convergence polls were obviously fine).

    The other point is that the Yes vote would probably have been much lower in most polls if there had been weighting by 2010 vote recall, as Panelbase have done in this poll. So the comparison you're making is not entirely meaningful.

  7. This blog is dying. Time it was put out of its misery.


    Na, looking good anon.

  9. Apologies if this is a stupid question - what effect does the different electorates for Westminster and Holyrood (eg EU citizens don't vote for Westminster) have?
    Obviously if someone couldn't vote in 2010 they shouldn't have a recalled vote, but on another polling site they were getting mildly worked-up about a large number of "did not vote at Westminster but did at Holyrood" in one poll, and I thought this was a simple explanation for some of the cases, which they'd missed. I was too late to point it out.