Thursday, August 14, 2014

There's just one thing I don't understand, Professor

See if you can reconcile two statements from Professor John Curtice, made within the same blogpost about the new female-only Survation poll. I must say I'm struggling.

Statement 1 : "In the company’s last regular monthly omnibus for the Daily Record in July, Yes support amongst women stood at 40%, exactly the same as today’s poll. Meanwhile the equivalent figures for the three months before that were June, 41%, May 38%, and April just 36%."

What he's pointing out here is that support for Yes among women in this poll is roughly the same as it was in two previous Survation polls which had Yes at 47% among the entire sample, and is significantly higher than in two earlier polls that had Yes at 44-45%. In other words, although it's impossible to draw firm conclusions, the new poll is perfectly consistent with Yes being above 45% across the population as a whole.

Statement 2 : "Of course what the poll does do is to add to the substantial weight of evidence that the Yes side is well behind in the polls."

Er, come again? How does a poll which could easily be consistent with Yes being as high as 47% add to the weight of evidence that Yes is "well" behind? Or does a 47/53 split suddenly fit the definition of a "commanding" lead, to use Survation's recent word of choice?

In all honesty, though, I was shaking my head in disbelief looking through Survation's datasets. Respondents in the South of Scotland have been upweighted from 55 to 128, and over-65s have been upweighted from 95 to 232. The upweighting of young voters isn't quite so extreme this time, but it's still substantial. All of this introduces a level of randomness (and likely volatility) into Survation's results which just isn't satisfactory. I get the slight impression that the various upweightings may have slightly helped Yes on this occasion, but it's difficult to be sure given that it's a female-only poll. Yet again, the No lead is fractionally lower in the raw unweighted data than it is in the weighted numbers, but that's probably due in large part to the extreme shortage of older voters in the raw sample.

To the extent that we can take the results seriously, though, the good news is that the gap is slightly smaller on the unrounded numbers...

Should Scotland be an independent country? (Women only, Don't Knows excluded)

Yes 40.4% (+1.3)
No 59.6% (-1.3)

Undecided female voters were pressed on how they would be most likely to vote, and they split 48.7% Yes, 51.3% No (excluding respondents who still couldn't give a view). Because that's a much narrower gap than in the headline numbers, it means that the No lead would be slightly lower if 'undecided leaners' were added in. For technical reasons, it's not possible to make the calculation, but it would probably work out as either a 41/59 or 42/58 split (and based on Curtice's observation, that could easily be consistent with a Yes vote of as high as 48% once men are taken into account!).

One thing I should have pointed out last night is that Survation still haven't joined the new orthodoxy of weighting by country of birth, so for that reason they may well be understating the Yes vote slightly in all of their polls.


  1. Someone posted on another blog that the Prof is on a retainer from the BBC/British State. I think someone asked him if this was true on his blog, but he never answered.

  2. All I can say is that it's quite cheeky that the Prof's department at Strathclyde teaches students how to put together objective surveys but, when it comes down to it, he doesn't bother to do that himself. If true that he is 'on a retainer' this should be outlawed, same as there is no need for MPs to have second jobs - they already have a full time job they are meant to be doing, and well recompensed they are for it as well.

  3. I think you worry too much about upweighting by age or region. It's true that it increases margins of error a little but not to do so risks bringing in a much greater bias. And some groups are just too hard to reach to get a decent sample - notoriously DE men under 25 for example.

    The real worry here is that Survation have only weighted for age and region which seems odd. They asked 2011 Holyrood vote but haven't weighted for it nor for SEG, never mind the newbie country of birth (onlie begetter scottish_skier)

    I assume this is because they don't have a definitive gender breakdown for 2011 vote, though you would think something could be calculated. Based on YouGov's poll of those who voted in 2011 (taken that evening), which I've adjusted to allow for YouGov's sample being generally under for SNP, I would reckon that the women's (constituency) vote split in 2011 went:

    SNP 43%

    Lab 32%

    Con 15%

    LD 10%

    (other parties only got 3% at most which I've split)

    Survation's sample here is 46/25/16/13. Labour under-representation is mostly compensated by other No parties, though the SP polling will be affected.

    Even more surprising is not weighting the sample by socio-economic group, because there is also a link with referendum preference there. The figures by gender should be derivable from the 2011 Census and at glance DE seem under-represented by quite a bit in this sample.