Thursday, September 11, 2014

Survation appear to be showing Yes on 47.1% - a slight increase on the last poll

Apologies for taking so long to get some analysis of the Survation poll up.  I was out and about today (in fact I passed Blair Jenkins as he was doing a TV interview on the Mound!) and I was checking my mobile phone so much to find out what was going on with the poll that I used up my battery.  Anyway, as usual I'll update this post gradually with my thoughts.

The first thing to say tonight is that Survation's Damian Lyons Lowe has not exactly covered himself in glory - he's the boy who cried wolf, and people will quite rightly be sceptical of what he says in future if he ever again attempts to hype up a poll in advance.  His suggestion overnight that this poll was "quite something!" could reasonably be interpreted as meaning one of two things - either a) a big swing to Yes, or b) a big swing to No.  There is absolutely no way of reconciling it with the reality of what the figures show, which appears to be a very, very small swing to Yes.  The comment was nothing more than a tiresome marketing ploy, and one that was plainly intended to mislead.  I should add of course that this isn't a bloody game - the long-term future of our country is hanging in the balance, nerves are absolutely shredded to such a point that people are practically making themselves ill, and this kind of attention-seeking stunt from a pollster is really not helping anyone.

As far as I can see, Survation have so far only released a small part of their datasets for this poll.  That's left me in difficulty trying to make much sense of what I'm looking at, because normally Survation's datasets include two sets of voting intention figures - one filtered for turnout, and one weighted for turnout - with the latter being the headline numbers.  There's only one set given in what's been released so far, and it's not at all clear which one it is.  Either way, though, it shows the Yes vote edging up very slightly on the last Survation poll...

Should Scotland be an independent country?

Yes 47.1% (+0.5 or +0.2)
No 52.9% (-0.5 or -0.2)

But whether those can be definitively regarded as the headline figures is beyond me at this stage.

As always with Survation, the devil is in the data.  There are two fundamental and ongoing problems with their Scottish panel that they seem completely unable or unwilling to resolve - they have far too few people under the age of 25, and far too few people who live in the South of Scotland electoral region.  The practical effect is that in poll after poll, they have to upweight the small number of respondents they do have in those two demographic groups by a huge amount, thus magnifying the effect of the random sampling variation that is bound to occur when you are dealing with such a small number of respondents.  In theory, that should lead to a greater amount of volatility.  Oddly enough, that hasn't happened (with the big exception of the phantom 'No bounce' after the first debate that no other firm detected), but what it certainly leads to is reduced reliability.  A claimed margin of error of 3% is fairly meaningless when this kind of extreme upweighting is going on.

Regular readers will probably remember that this was precisely the reason why I was a touch dubious about all of the first three Survation polls that put Yes on 47%.  The first two had particularly favourable figures for Yes among the tiny sample of under-25s, who had been upweighted roughly three-fold, while the third poll saw implausibly wonderful figures for Yes among the South of Scotland sample, which had been upweighted two-fold.  So when Yes merely "recovered" to 47% in the first Survation poll to be conducted after the second debate, I wondered if it was in real terms the best ever Yes showing with the firm, because it was the first time 47% had been reached with normal-looking figures among young people and in the south.  Tonight's poll takes that trend a bit further, because this time Yes have stayed on 47% in spite of figures for young people that are positively dreadful.  Of the tiny sample of 57 respondents under the age of 25, no fewer than 51% are voting No and only 32% are voting Yes.  Those 57 people have been upweighted to count as 130 'virtual' respondents, meaning that Survation's weighting procedure has created approximately 37 young No voters out of thin air.  I know of no good reason to think that there is a 19-point No lead among under-25s (both the YouGov and TNS polls showed an outright Yes lead in that age group), so this looks like a pretty clear-cut case of sampling variation leading to an underestimation of the Yes vote.  And that hasn't been offset in any way by the results from the heavily upweighted sample of voters from the south - in fact the No lead in that sample has grown even more since the last poll.

What I've given above is a reason for slightly distrusting both the headline numbers in this poll and the trend.  There's also an additional reason for distrusting Survation's headline numbers, but which doesn't have any relevance to the trend they show, and that's their continuing failure to join the new orthodoxy of weighting by country of birth.  We don't actually know for a fact that Survation have too many English-born respondents in their samples, but given that it's a problem that has been clearly established among all of the other three online pollsters, it doesn't seem far-fetched to think that it might apply to Survation as well.  If it does, an adjustment to correct the error might well reduce the No lead by 2% or so.  That may not seem like much, but in the light of the potential closeness of this contest, it's absolutely crucial, and the lack of any adjustment may be distorting our perception of the state of play.  Incidentally, that observation may well also apply to the poll expected this coming weekend from ICM, who have similarly refused so far to introduce country of birth weighting in spite of having already established that they have considerably too many English-born respondents in their samples.

Last night, I stumbled upon someone live-tweeting a lecture from Professor Curtice, who was apparently saying that "something" was definitely happening, but it was unclear whether that something was a swing, or convergence between pollsters - and that the Survation poll would help to solve the puzzle.  Presumably the verdict is now going to be convergence, but surely it's self-evident that's a false choice.  Convergence couldn't possibly be happening unless a genuine swing had taken place.  If the reason for the disagreements about the recent trend is that different pollsters have been measuring something slightly different to each other, that would suggest a big swing has occurred among the type of people TNS and YouGov interview, but not among the type of people Panelbase and Survation interview (who were proportionately more likely to be Yes voters in the first place).  Therefore, the most plausible conclusion to draw is that the position on the ground is somewhere in the middle - there has been a swing to Yes, but not quite as big a one as suggested by YouGov and TNS.

However, there's a caveat about the whole convergence theory, because Survation's contribution to it depends on the assumption that the new poll is directly comparable to the TNS and YouGov polls, which as a result of the fieldwork dates is not fully the case.  Survation's fieldwork can be split into two distinct periods - Friday and Saturday, before the pandemonium started when YouGov gave Yes a slight lead, and Sunday to Tuesday.  There are three possibilities of what has happened -

1) There was a swing to Yes in Survation's fieldwork on Friday and Saturday, in line with what YouGov and TNS showed.  That would mean there must have been a significant drop in Yes support from Sunday onwards in order for the results to average out as unchanged.  If so, we can expect a significant swing to No in tomorrow night's YouGov poll.

2) There was an absolutely steady picture across the five days of Survation's fieldwork.  That would be wholly consistent with Curtice's convergence theory, but paradoxically would mean that all bets are off for tomorrow night's YouGov.  We've got used to the headline numbers of different firms diverging in this campaign, but until now the trends have often been broadly similar - if that's no longer going to be the case, then predictions are utterly impossible.

3) There was actually a fall in Yes support in the fieldwork from Friday and Saturday, which has been offset by a Yes bandwagon effect from Sunday onwards.  This is the least likely of the three possibilities (because evidence from other pollsters contradicts the idea that Yes were slipping prior to Sunday), but the problems with Survation's sampling means that it shouldn't be dismissed out of hand.  And if that is what's happened, we might expect a swing to Yes in the YouGov poll.

That said, you'd probably expect some kind of reduction in the Yes vote in the YouGov poll, if only due to a reversion to the mean after the almost unbelievably big swing to Yes that has occurred over the last month.  The greater the reduction is, the less likely it is to be caused purely by margin of error noise - although even slippage to 45% or 46% could be just about explained away by sampling variation.

* * *


Swing required for 1 out of 6 pollsters to show Yes ahead or level : 0.0%

Swing required for 2 out of 6 pollsters to show Yes ahead or level : 0.5%

Swing required for 3 out of 6 pollsters to show Yes ahead or level : 2.0%

Swing required for 4 out of 6 pollsters to show Yes ahead or level : 3.0%

Swing required for 5 out of 6 pollsters to show Yes ahead or level : 4.5%

Swing required for 6 out of 6 pollsters to show Yes ahead or level : 7.0%

* * *


MEAN AVERAGE (excluding Don't Knows) :

Yes 47.0% (n/c)
No 53.0% (n/c)

MEAN AVERAGE (not excluding Don't Knows) :

Yes 41.5% (n/c)
No 46.8% (n/c)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 47.2% (n/c)
No 52.8% (n/c)

(The Poll of Polls is based on a rolling average of the most recent poll from each of the pollsters that have been active in the referendum campaign since September 2013, and that adhere to British Polling Council rules. At present, there are six - YouGov, TNS-BMRB, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample. Changes in the Poll of Polls are generally glacial in nature due to the fact that only a small portion of the sample is updated each time.)


  1. I was quite spooked by YouGov, because the result seemed to call all the hounds of hell onto our heads. Obviously part of me would love to see Yes above 55%, but on the other hand the Westminster machine is so vicious that the position of underdog has its attractions.

  2. Many thanks James for the clear analysis. It's exactly for this at this time that we crowdfunded your appeal.

  3. Standard survation pattens (although other polls show the same) of lowest Yes / Highest No where they've struggled to get respondents.

  4. As a YES activist, I'm finding that there is a increasing distrust of the polls and even more distrust of the way the media reports it (and everything else).

    I recommend your site to anyone who has expressed frustration at the media for a more reasoned look at the detail.

    Keep up the good work.

  5. Just thinking about the past polls, it still strikes me as amazing (if youre looking at things objectively) that Survation did recently did a full scale poll of the age group where they had the most respondents, and where voting patterns are stupidly stable. What more information did this add?

    Surely it would have been more useful to do a poll of the most volatile age group, i.e. the 16-24 year olds.

    But then it all depends on who pays the bills I guess.

  6. Here in Australia there is a ban on all opinion polls from the Wednesday before election day (always a Saturday) until the actual close of polling.

    Might be a good idea for an independent Scotland to consider this, if only for the simple reason I don't think my nerves can stand this for much longer !

  7. Yougov UK Scotland subset:

    How likely or unlikely do you think it is that Scotland will vote to become independent in the referendum this month?

    51% Likely
    35% Unlikely

  8. Seems to me that the divergence of the various polls shows conclusively that some of them are wrong, and has done for some time.

    This latest set of polls seems to show that it is the most no friendly polls which have been most wrong.

    Am I missing something obvious?

  9. "this isn't a bloody game"

    well said James.

    Nerves = shredded already. :-(

    thanks for the calm analysis though.

  10. Just listened to Survation being interviewed ,making a lot about the gender gap and the age gap ,how YES has stagnated ,All pretty dodgy stuff based on this last poll
    It looks as if this poll has made a major flaw in its up weighting of its atypical sampling of the 16-24 group unfortunately the media frenzy resulting from this is incredible ,I see the odd for a yes vote has lengthened as well
    I await the polls due this weekend with interest
    I am anticipating a normalising back to a small Yes lead
    Watch for more media frenzy

  11. Apparently, there'll be a ICM poll out tomorrow. The poll has been conducted through telephone between Tuesday and Thursday this week. It is for the Guardian. Patrick Wintour tweeted.


  12. This comment has been removed by the author.

  13. This comment has been removed by the author.

  14. Given the time frame of this incoming ICM poll (that finished asking today), can we expect the figures to be leaked tonight? Or do you think that they are referring to some point tomorrow?


  15. Can't help but feel powerless with all these polls producing [dubious?] results. The people of Scotland have rightly seen through the bias of the media, I hope they view these polls with similar scepticism.

  16. Alberto Nardelli ‏@AlbertoNardelli 4m
    Guardian/ICM Scotland #indyref poll coming soon

    Mike Smithson @MSmithsonPB · 2h
    Looks as though there'll be a new YouGov IndyRef poll out later this evening for tomorrow's Times/Sun. Will firm still have YES ahead?

  17. As a matter for info 97% of eligible electorate have registered to vote in this referendum.

    Headache for pollsters?

  18. Perhaps. In Quebec that type of massive turnout put the final polls out by 4 to 5% and it was no they underestimated. Anyone want to guess if it would be the same here? I suspect it going the other way is more likely on that turnout.