Unveiling the Scot Goes Pop Poll of Polls for the EU referendum

I know this falls into the category of "he would say that, wouldn't he", but I always felt that the Poll of Polls I ran on this blog during the independence referendum campaign gave a much more meaningful indication of the direction of travel than John Curtice's Poll of Polls. The Curtice version was an average of the most recent few polls, regardless of which firms had conducted them, or what methodology had been used. That meant it sometimes gave the illusory impression that Yes or No were making progress simply because the newest update included a larger number of polls from Yes-friendly or No-friendly firms. By contrast, the Scot Goes Pop Poll of Polls only included the most recent poll from each firm, meaning that any statistically significant changes were more likely to reflect real movement on the ground.

Professor Curtice is sticking to his preferred method for his new EU referendum Poll of Polls, and if anything that's going to create even bigger problems this time, because the gulf between telephone and online polls is absolutely enormous at the moment. Telephone firms are suggesting there is a hefty lead for 'Remain', while online firms think it's either a statistical tie, or very close to that. Online polls are likely to appear more frequently, because they're cheaper to conduct, so any Poll of Polls that gives equal weight to all recent polls is automatically giving greater weight to the 'Leave'-friendly online data collection method. In fact, the current update of the Curtice PoP is almost entirely comprised of online polls, producing a result of Remain 54%, Leave 46% (with Don't Knows excluded). Unless there is some rational reason for concluding at this early stage that the online method is far more reliable (and there isn't), those figures don't meaningfully reflect the totality of current polling evidence. And because the exact weighting given to online polls will vary from update to update (sometimes it might rise to 100%, occasionally it might drop to 50%), future results will fluctuate in an almost random way.

So I thought it might be a good idea to once again run a Poll of Polls here that is designed to be better at picking up the trend. I'm not going to replicate the method I used for the independence referendum, because I think that might get too complicated - some firms will be regularly conducting both telephone and online polls, so I would have to decide whether to count "ICM (telephone)" and "ICM (online)" as separate firms. Instead, I'm going to produce three averages - one for all telephone polls conducted at least partly within the last month, one for all online polls conducted at least partly within the last month, and a composite 'headline' set of figures that give an equal 50/50 weighting to telephone and online. Obviously the latter average will be a bit artificial, but at least this way we'll get a sense of what the real state of play is if telephone polls are correct, or if online polls are correct, or if both types of poll are wrong by roughly an equal amount. Under the current circumstances, I'm not sure it's possible to do a whole lot better than that.

Should the United Kingdom remain a member of the European Union or leave the European Union?

50/50 ONLINE/TELEPHONE AVERAGE :

Remain 48.7%
Leave 37.2%

ONLINE AVERAGE :

Remain 43.8%
Leave 38.3%

TELEPHONE AVERAGE :

Remain 53.5%
Leave 36.0%

(The online average is based on six polls - five from ICM and one from YouGov. The telephone average is based on one poll from ComRes and one from Ipsos-Mori.)

As big as the gap between the two component averages looks, it could easily have been much bigger - the new telephone poll from Ipsos-Mori caused a surprise by showing a slump in the 'Remain' lead. That can't be explained away simply by the introduction of the new question, because they also asked exactly the same question that was posed in the firm's previous poll in June, producing much the same result. It could be that the swing is being exaggerated due to normal sampling variation, but unless either this poll or the last one was an outright rogue, it seems likely that it's picking up some kind of genuine shift.