Reasons to be sceptical about the changes in the new Survation poll

I've been away all day, but I've finally had a chance to catch up with the way Survation are reporting the findings of their own poll, and quite frankly it's laughable.  It's obviously a matter of perception whether the No campaign can be reasonably said to have a "commanding" lead when Yes are on 43% - just 7% from victory.  But one thing that is undeniable is that it's a much lower lead than in countless polls we've seen in this campaign, especially up until a few months ago.

There's one particular line that contains a downright inaccuracy (I'm tempted to be a good bit more blunt than that) -

"This is the highest ‘No’ vote – and the biggest lead over ‘Yes’ – seen in a Survation poll since our first independence referendum poll in February."

The "first" poll they are talking about there was in fact their second - they are attempting to erase from history their first referendum poll in January which showed a much, much bigger No lead.  It's hard not to suspect that they are doing so to please their clients the Daily Mail, who were plainly keen to make the false claim that the leaders' debate had led to a record lead for No.  What possible justification can there be to pretend the January poll didn't happen?  Well, Survation have radically altered their methodology since then, and it's impossible to know for sure what the outcome of that poll would have been under the current procedures.  But it is possible to hazard an educated guess based on the raw unweighted data, which as it happens suggests that the No lead would have been slightly higher in January than it was in last night's poll.

The breathless way in which Survation report their finding that Darling "won" the debate verges on the idiotic.  The only credible way to gauge the winner of a debate is to conduct an instant poll immediately afterwards - if you wait until a day or two later (as Survation did), public perceptions will be hopelessly tainted by the media spin on who "won".  But hats off to the impressively high 28.3% of Survation's respondents who know their own mind, and who defied the media narrative by saying Salmond won the debate.

To turn to the headline voting intention results, there are two specific reasons to be sceptical about the supposed swing to No (over and above the possibility that it's largely an illusion caused by the standard 3% margin of error).  There are two groups of respondents who showed particularly big swings to No, and who had to be upweighted massively because Survation couldn't find enough people in those demographic categories.  Whenever that happens, it effectively increases the margin of error for the overall sample, and makes volatility from one poll to the next much more probable.  Indeed, I seem to recall saying in a post not too long ago that Survation's constant need to drastically upweight the youngest group of respondents really ought to be causing volatility, and it was surprising that it wasn't.  That has finally happened now, because since last week's poll the No lead among 16-24 year olds has increased from 5.1% to 34.6%.  The effect of that wildly implausible swing has been magnified in the overall results, because the 61 real respondents in that age group have been upweighted more than two-fold to count as 129 'virtual' respondents.  Self-evidently, there hasn't really been a 15% swing to No among young people in the space of one week, so it's likely that the weighting procedure has artificially generated at least part of the increase in the overall No lead.  Indeed, if we look at the other age groups (who haven't needed to be upweighted significantly), it's striking how similar this week's poll is to last week's - No have gained a little ground among three age groups, and Yes have gained a little ground among two.  But nothing of any great statistical significance.

The other part of the datasets where alarm bells start ringing is the regional breakdown for the South of Scotland.  Last week, Yes had an 11% lead in that region, whereas now they are behind by 24%.  Again, the impact of that turnaround has been magnified, because in both polls respondents from the South had to be upweighted roughly two-fold.  Now, to be fair, this week's results are actually more plausible than last week's - you'd expect the South to be one of the most No-friendly regions.  But the point is that the extreme upweighting of two such different figures has distorted the trend.

The best clue that No have been flattered in this poll lies in the unweighted data.  In the vast majority of polls, the weighting helps Yes, but in this one the No lead is slightly smaller on the unweighted figures.  Indeed, the No lead on the unweighted figures of this poll is smaller than the No lead was on the unweighted figures of the last-but-one poll from Survation.

What I draw from all of this is a) it's far from clear at this stage that there has been ANY post-debate bounce for No at all, and b) if there has been a bounce, the balance of probability is that it's smaller than Survation's headline numbers would suggest.

*  *  *

This supplementary question made me laugh -

"Do you believe the Scottish Government should draw up alternative options to a ‘currency union’ ahead of the referendum on September 18, 2014?"

What, like they did several months ago, in rather a lot of detail?  In next week's Survation poll -

"Do you think Clement Attlee would have been a better Prime Minister if he'd introduced a National Health Service?  Why oh why didn't he do it?"