Survation appear to be showing Yes on 47.1% - a slight increase on the last poll

Apologies for taking so long to get some analysis of the Survation poll up.  I was out and about today (in fact I passed Blair Jenkins as he was doing a TV interview on the Mound!) and I was checking my mobile phone so much to find out what was going on with the poll that I used up my battery.  Anyway, as usual I'll update this post gradually with my thoughts.

The first thing to say tonight is that Survation's Damian Lyons Lowe has not exactly covered himself in glory - he's the boy who cried wolf, and people will quite rightly be sceptical of what he says in future if he ever again attempts to hype up a poll in advance.  His suggestion overnight that this poll was "quite something!" could reasonably be interpreted as meaning one of two things - either a) a big swing to Yes, or b) a big swing to No.  There is absolutely no way of reconciling it with the reality of what the figures show, which appears to be a very, very small swing to Yes.  The comment was nothing more than a tiresome marketing ploy, and one that was plainly intended to mislead.  I should add of course that this isn't a bloody game - the long-term future of our country is hanging in the balance, nerves are absolutely shredded to such a point that people are practically making themselves ill, and this kind of attention-seeking stunt from a pollster is really not helping anyone.

As far as I can see, Survation have so far only released a small part of their datasets for this poll.  That's left me in difficulty trying to make much sense of what I'm looking at, because normally Survation's datasets include two sets of voting intention figures - one filtered for turnout, and one weighted for turnout - with the latter being the headline numbers.  There's only one set given in what's been released so far, and it's not at all clear which one it is.  Either way, though, it shows the Yes vote edging up very slightly on the last Survation poll...

Should Scotland be an independent country?

Yes 47.1% (+0.5 or +0.2)
No 52.9% (-0.5 or -0.2)

But whether those can be definitively regarded as the headline figures is beyond me at this stage.

As always with Survation, the devil is in the data.  There are two fundamental and ongoing problems with their Scottish panel that they seem completely unable or unwilling to resolve - they have far too few people under the age of 25, and far too few people who live in the South of Scotland electoral region.  The practical effect is that in poll after poll, they have to upweight the small number of respondents they do have in those two demographic groups by a huge amount, thus magnifying the effect of the random sampling variation that is bound to occur when you are dealing with such a small number of respondents.  In theory, that should lead to a greater amount of volatility.  Oddly enough, that hasn't happened (with the big exception of the phantom 'No bounce' after the first debate that no other firm detected), but what it certainly leads to is reduced reliability.  A claimed margin of error of 3% is fairly meaningless when this kind of extreme upweighting is going on.

Regular readers will probably remember that this was precisely the reason why I was a touch dubious about all of the first three Survation polls that put Yes on 47%.  The first two had particularly favourable figures for Yes among the tiny sample of under-25s, who had been upweighted roughly three-fold, while the third poll saw implausibly wonderful figures for Yes among the South of Scotland sample, which had been upweighted two-fold.  So when Yes merely "recovered" to 47% in the first Survation poll to be conducted after the second debate, I wondered if it was in real terms the best ever Yes showing with the firm, because it was the first time 47% had been reached with normal-looking figures among young people and in the south.  Tonight's poll takes that trend a bit further, because this time Yes have stayed on 47% in spite of figures for young people that are positively dreadful.  Of the tiny sample of 57 respondents under the age of 25, no fewer than 51% are voting No and only 32% are voting Yes.  Those 57 people have been upweighted to count as 130 'virtual' respondents, meaning that Survation's weighting procedure has created approximately 37 young No voters out of thin air.  I know of no good reason to think that there is a 19-point No lead among under-25s (both the YouGov and TNS polls showed an outright Yes lead in that age group), so this looks like a pretty clear-cut case of sampling variation leading to an underestimation of the Yes vote.  And that hasn't been offset in any way by the results from the heavily upweighted sample of voters from the south - in fact the No lead in that sample has grown even more since the last poll.

What I've given above is a reason for slightly distrusting both the headline numbers in this poll and the trend.  There's also an additional reason for distrusting Survation's headline numbers, but which doesn't have any relevance to the trend they show, and that's their continuing failure to join the new orthodoxy of weighting by country of birth.  We don't actually know for a fact that Survation have too many English-born respondents in their samples, but given that it's a problem that has been clearly established among all of the other three online pollsters, it doesn't seem far-fetched to think that it might apply to Survation as well.  If it does, an adjustment to correct the error might well reduce the No lead by 2% or so.  That may not seem like much, but in the light of the potential closeness of this contest, it's absolutely crucial, and the lack of any adjustment may be distorting our perception of the state of play.  Incidentally, that observation may well also apply to the poll expected this coming weekend from ICM, who have similarly refused so far to introduce country of birth weighting in spite of having already established that they have considerably too many English-born respondents in their samples.

Last night, I stumbled upon someone live-tweeting a lecture from Professor Curtice, who was apparently saying that "something" was definitely happening, but it was unclear whether that something was a swing, or convergence between pollsters - and that the Survation poll would help to solve the puzzle.  Presumably the verdict is now going to be convergence, but surely it's self-evident that's a false choice.  Convergence couldn't possibly be happening unless a genuine swing had taken place.  If the reason for the disagreements about the recent trend is that different pollsters have been measuring something slightly different to each other, that would suggest a big swing has occurred among the type of people TNS and YouGov interview, but not among the type of people Panelbase and Survation interview (who were proportionately more likely to be Yes voters in the first place).  Therefore, the most plausible conclusion to draw is that the position on the ground is somewhere in the middle - there has been a swing to Yes, but not quite as big a one as suggested by YouGov and TNS.

However, there's a caveat about the whole convergence theory, because Survation's contribution to it depends on the assumption that the new poll is directly comparable to the TNS and YouGov polls, which as a result of the fieldwork dates is not fully the case.  Survation's fieldwork can be split into two distinct periods - Friday and Saturday, before the pandemonium started when YouGov gave Yes a slight lead, and Sunday to Tuesday.  There are three possibilities of what has happened -

1) There was a swing to Yes in Survation's fieldwork on Friday and Saturday, in line with what YouGov and TNS showed.  That would mean there must have been a significant drop in Yes support from Sunday onwards in order for the results to average out as unchanged.  If so, we can expect a significant swing to No in tomorrow night's YouGov poll.

2) There was an absolutely steady picture across the five days of Survation's fieldwork.  That would be wholly consistent with Curtice's convergence theory, but paradoxically would mean that all bets are off for tomorrow night's YouGov.  We've got used to the headline numbers of different firms diverging in this campaign, but until now the trends have often been broadly similar - if that's no longer going to be the case, then predictions are utterly impossible.

3) There was actually a fall in Yes support in the fieldwork from Friday and Saturday, which has been offset by a Yes bandwagon effect from Sunday onwards.  This is the least likely of the three possibilities (because evidence from other pollsters contradicts the idea that Yes were slipping prior to Sunday), but the problems with Survation's sampling means that it shouldn't be dismissed out of hand.  And if that is what's happened, we might expect a swing to Yes in the YouGov poll.

That said, you'd probably expect some kind of reduction in the Yes vote in the YouGov poll, if only due to a reversion to the mean after the almost unbelievably big swing to Yes that has occurred over the last month.  The greater the reduction is, the less likely it is to be caused purely by margin of error noise - although even slippage to 45% or 46% could be just about explained away by sampling variation.

* * *

REQUIRED SWINGS

Swing required for 1 out of 6 pollsters to show Yes ahead or level : 0.0%

Swing required for 2 out of 6 pollsters to show Yes ahead or level : 0.5%

Swing required for 3 out of 6 pollsters to show Yes ahead or level : 2.0%

Swing required for 4 out of 6 pollsters to show Yes ahead or level : 3.0%

Swing required for 5 out of 6 pollsters to show Yes ahead or level : 4.5%

Swing required for 6 out of 6 pollsters to show Yes ahead or level : 7.0%

* * *

SCOT GOES POP POLL OF POLLS

MEAN AVERAGE (excluding Don't Knows) :

Yes 47.0% (n/c)
No 53.0% (n/c)

MEAN AVERAGE (not excluding Don't Knows) :

Yes 41.5% (n/c)
No 46.8% (n/c)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 47.2% (n/c)
No 52.8% (n/c)

(The Poll of Polls is based on a rolling average of the most recent poll from each of the pollsters that have been active in the referendum campaign since September 2013, and that adhere to British Polling Council rules. At present, there are six - YouGov, TNS-BMRB, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample. Changes in the Poll of Polls are generally glacial in nature due to the fact that only a small portion of the sample is updated each time.)