The distorting average

I know that a number of us feel that John Curtice's polling analysis is now noticeably fairer to the Yes campaign than it was back in the summer and autumn. However, this pronouncement from him today really bugs me, because it makes no logical sense whatever -

"Now that Panelbase have reported, we have had at least one post-White Paper reading of the state of the referendum race from all those companies that have been polling referendum vote intentions on a reasonably regular basis. That means we can finally make a reasonably robust estimate of where the polls stand on average as compared with the position up to the end of November. In the dozen polls conducted wholly or mostly between September and November the Yes vote averaged 38% (after the exclusion of the Don’t Knows). In the nine polls that have been conducted wholly or mostly since the beginning of August the Yes tally has averaged 40%. The Yes side may have made some progress in the last couple of months, but it evidently still some considerable distance away from the winning post."

Even if we take those numbers at face value, they're clearly moderately encouraging for the Yes campaign. But are they telling us the whole story? First of all (on a point of pedantry), even counting the non-BPC Progressive Scottish Opinion, I simply cannot find twelve polls that were conducted wholly or mostly between September and November. But of the ten that were, this is the breakdown -

Panelbase (Yes-friendly) 3 polls
TNS-BMRB (mover) 3 polls
Progressive Scottish Opinion (No-friendly) 1 poll
Ipsos-Mori (No-friendly) 1 poll
ICM (mover) 1 poll
YouGov (No-friendly) 1 poll

TOTAL YES-FRIENDLY - 30%
TOTAL NO-FRIENDLY - 30%

And of the nine polls that have been conducted since the start of December (presumably that's what Curtice meant rather than the start of August), this is the breakdown -

YouGov (No-friendly) 3 polls
TNS-BMRB (mover) 2 polls
ICM (mover) 1 poll
Survation (No-friendly) 1 poll
Ipsos-Mori (No-friendly) 1 poll
Panelbase (Yes-friendly) 1 poll

TOTAL YES-FRIENDLY - 11%
TOTAL NO-FRIENDLY - 56%

So comparing the crude polling averages of the two periods is the equivalent of comparing apples with oranges.  The September-November period contained a much greater proportion of polls from Panelbase, which has persistently been a much more favourable pollster for the Yes campaign than YouGov, which in turn contributed a lower proportion of polls in the same period.  Although it isn't possible to project what the figures would be if the relative contributions of each pollster had been the same in both periods, it's virtually certain that the average Yes vote for December-February would be higher than 40% if, in line with the earlier period, there had been a greater number of Panelbase polls and a smaller number of YouGov polls - meaning by extension that the average increase in the Yes vote would probably be higher than 2%. 

If that seems a counterintuitive point in the light of the trend in today's poll, bear in mind that, notwithstanding the recent narrowing of the differential between YouGov and Panelbase, the former are currently showing a Yes vote of 39% with undecideds excluded, while the latter are still showing a Yes vote of 43%.  That, in a nutshell, is the reason why this blog's Poll of Polls only takes account of the most recent poll from each pollster - it's the only way we can be sure of like-for-like comparisons and meaningful trends.  It's not a perfect system by any means - for one thing I'll have a dilemma about how long to keep taking account of Angus Reid's most recent poll from the summer if they don't conduct a new one soon.  But with the best will in the world, I just don't see how the conventional approach of crudely averaging all polls within a set period is going to work in a campaign in which different pollsters are producing headline figures that are light-years apart.

* * *

Although I still haven't seen the datasets for today's Panelbase poll, it seems to be the case that the SNP's Holyrood support has gone back up to landslide territory.  That's very good news in more ways than one, because it reinforces the point that there was nothing suspicious about the whopping SNP vote suggested by the unweighted data in last week's Survation poll, and that there was therefore no conceivable rationale for downweighting both the SNP and Yes votes on the industrial scale that happened.  It also means that there was nothing suspicious about the 19-point SNP lead in European Parliament voting intentions found by the ICM poll that showed a big swing to Yes.