Sunday, May 18, 2014

Will Andrew Marr please stop inventing his own facts?

There is a section of the BBC's guidelines on impartial coverage of the referendum campaign which specifically deals with how opinion polls should be handled by presenters and journalists. To paraphrase what is said, polls should only be mentioned sparingly, and presenters should add a bit of context along the lines of "that's out of line with what we've been seeing recently" in order to ensure that an outlying poll is not given undue prominence. But Andrew Marr seems to have misinterpreted that as an instruction to invent an entirely fictional "context" in order to hoodwink his viewers into thinking that an outlying trend shown by a single poll is in fact typical of other polls. I gather he mentioned the ICM poll in his review of the papers this morning and claimed that a lot of polls had been showing the same thing recently. Let's look at the evidence, shall we? Other than the ICM poll, this is what we've had since Easter -

Survation have shown the No lead dropping from 10 points to 8, followed by a return to the status quo ante of 10 points. So no change there at all.

YouGov have shown the No lead dropping from 15 points to a new low for the campaign of 14 points. Yes, that poll was widely misreported as being good for the No campaign on the basis of an 'apples and oranges' comparison with the numbers from other pollsters, but Marr has a duty as a public service TV presenter to look at the facts, not the spin.

Panelbase have shown the No lead increasing from 5 points to 7 - a tiny increase that could very easily be (and probably is) margin of error 'noise'.

Progressive Scottish Opinion have shown the No lead plummeting from 29 points to 20, albeit with a very long gap between the two polls in question. The No campaign's propaganda has tried to con people into thinking that the new Progressive poll should for some reason be compared with the last YouGov poll rather than the previous poll by the same company, but again, Marr has a duty to cut through the propaganda and look at the facts.

TNS-BMRB have shown no change in the headline No lead of 12 points in their most recent poll, but the raw Yes vote is at its highest level since a major methodological change was introduced several months ago, and on the unrounded figures the No lead has fallen (albeit fractionally) to its lowest level of the campaign so far.

If Marr is aware of all that and still thinks he can justify his claim this morning, then evidently the laws of arithmetic must function very differently in his reality.

* * *

The datasets for the Panelbase poll are up already. It's amazing - they used to be very slow compared to other pollsters, now they seem to be faster than anyone! The first thing to say is that the No lead on the unrounded numbers is somewhat lower at 6.5%, the increase in the No lead is just 0.8% (as opposed to the 2% misleadingly suggested by the rounded numbers), and the raw Yes vote has actually gone up. That makes it even more likely that the tiny increase in the No lead is just margin of error noise, and indeed that the ICM poll is an outlier (in terms of the trend, if not necessarily the headline numbers).

Should Scotland be an independent country? (Figures rounded to one decimal place.)

Yes 40.1% (+0.5)
No 46.6% (+1.3)

With Don't Knows excluded -

Yes 46.2% (-0.5)
No 53.8% (+0.5)

There is also a follow-up question demonstrating that undecided voters are currently leaning (albeit only narrowly) towards a Yes vote. If we "just for a bit of fun" add in their preferences to the overall figures, this is what we end up with after the small remaining number who wouldn't vote at all are excluded -

Yes 47.2%
No 52.8%

Yeah, but "panic over", right?

By the way, it seems highly suspicious to me that Panelbase have not added the preamble they used to the datasets.  Unless no preamble was used at all (unlikely), that may well mean that they've gone back to using their old, subtly biased preamble, probably due to the fact that the poll was commissioned by an anti-independence client.  Whether that's the case or not, such a lack of transparency from a BPC pollster is certainly highly regrettable.  [UPDATE : Calum Findlay took part in the poll and has confirmed the neutral preamble was used.]

John Curtice is in a privileged position, because unlike most of the rest of us he has already seen the unpublished ICM datasets. On the basis of that, he's made an observation that the Yes vote would have fallen further in the new ICM poll if it hadn't been for a methodological change that for the first time takes account of likelihood to vote. I must say that comment troubles me, because ICM have been constantly tweaking their methodology over the last few months, and if you're going to make a direct long-term comparison you really need to unravel all of those changes and take the methodology back to "manufacturer's settings", so to speak. Cherry-picking one particular change gives a misleading impression, because it's highly unlikely that all of the changes are working in favour of the Yes campaign.


  1. Let's make a comparison between different methodologies (with DK removed) shall we?

    2011 past vote:
    Panelbase Yes: 44% No: 56%
    Survation Yes: 44% No: 56%

    2010/2011 past vote:
    YouGov Yes: 42% No: 58%

    2011 behaviour:
    TNS BMRB Yes: 41% No: 59%
    ICM Yes: 40% No: 60%

  2. All figures are for the full sample. Panelbase figure should have been 46%.

  3. Unionist quisling of the day prize for bonkers Marr maybe?