Tuesday, March 4, 2014

Ipsos-Mori consolidate their status as the extreme outliers

One thing that we can certainly deduce from yesterday's Ipsos-Mori poll is that the Yes campaign are highly unlikely to go into polling day in September in the lead with that particular pollster. But the million dollar question is whether they actually need to do so. The gap between the No lead being shown by Ipsos-Mori and the average No lead being shown by the four online pollsters that have reported so far this year is a stonking 12.25%. In other words, even if the online pollsters end up pointing to a narrow Yes victory by September, the chances are that Ipsos-Mori will still be showing a double-digit No lead. They really have turned into the extreme outliers of this campaign.

That's not to say they are necessarily getting it wrong, of course. They are unique in being the only referendum pollster to be carrying out their fieldwork by telephone, and because of that factor it's hard to shake off the nagging doubt that they may be getting a more representative sample than the online pollsters (at the very least, there's no danger that they're interviewing the same people over and over again as YouGov and Panelbase do). Luckily, however, we do have one other non-online pollster to make a comparison with. TNS-BMRB conduct their fieldwork face-to-face, and in recent polls have been showing a markedly more favourable position for Yes than Ipsos-Mori. So the non-online status of Ipsos-Mori can't in itself explain why they produce such different results from everyone else. A more likely explanation is their ongoing refusal to weight their results by past Holyrood vote, which is now standard practice for most other BPC pollsters. It's also known that they use a 'stealth preamble' when asking the referendum question, but we can only guess how leading or biased that may be.

And whatever the virtues of telephone polling (or indeed face-to-face polling), we shouldn't forget that there's one very good reason for thinking that online polls may be producing more accurate results, namely that 'shy Yes' voters probably find it harder to admit their preference to a live interviewer than to a computer screen.

All the same, this huge degree of uncertainty as to the true state of play is incredibly frustrating and unsettling. If ICM (the UK's "gold standard" pollster) and their online polling cousins of Panelbase and Survation are correct, the Yes campaign are within relatively easy striking distance of victory and an independent Scotland in 2016 is a real prospect. But if Ipsos-Mori are correct, the challenge faced by Yes is bordering on the insurmountable. I know most of us firmly believe that the former scenario is far more likely, but it's scary to think that we simply won't know for sure until the real votes start coming in.

As ever, when pollsters are so far apart the only way of making sense of any individual poll is to look at the trend, and Ipsos-Mori have at least confirmed that the No lead remains three points lower than in the last two polls prior to the publication of the White Paper.

* * *

SCOT GOES POP POLL OF POLLS

Since the No vote is unchanged in the Ipsos-Mori poll, it's only the Yes vote that is affected slightly in the updated headline figures of this blog's Poll of Polls.

MEAN AVERAGE (not excluding Don't Knows) :

Yes 34.6% (-0.3)
No 48.9% (n/c)

MEAN AVERAGE (excluding Don't Knows) :

Yes 41.4% (-0.2)
No 58.6% (+0.2)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 42.0% (n/c)
No 58.0% (n/c)


(The Poll of Polls is based on a rolling average of the most recent poll from each of the pollsters that have been active in the referendum campaign, and that adhere to British Polling Council rules. At present, there are seven - Angus Reid, YouGov, TNS-BMRB, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample. Changes in the Poll of Polls are generally glacial in nature due to the fact that only a small portion of the sample is updated each time.

For clarity, the Poll of Polls takes no account of polls conducted by bridalwear companies.)


The fact that the median average has become more favourable to Yes than the mean average is testament to how the Yes-friendly end of the polling spectrum is now more heavily populated than the No-friendly end, a complete reversal of the position that held good until recently.

9 comments:

  1. Aren't they also weighting on 2010 GE results rather than 2011 Holyrood?

    ReplyDelete
  2. Sorry, I see you made that point already.

    ReplyDelete
  3. "it's scary to think that we simply won't know for sure until the real votes start coming in."

    I can see one benefit to yes in that though: some people may be a little unsure about independence, but have a preference for a lot more powers and have been turned off by the no campaign.

    If those people don't believe yes will win, but also feel a stonking no result will deliver nothing but an unbearably smug no and Westminster camp, they're more likely to vote yes in the hope a close result will bring them what they want.

    If the polls are actually skewed towards no, that could bring a yes victory.

    ReplyDelete
  4. Michael : No, they don't weight by past vote at all (neither Holyrood nor Westminster).

    ReplyDelete
  5. Duncan HotdogstallMarch 4, 2014 at 10:56 AM

    Hi James,

    A glance through the data tables is setting off some alarm bells. I'm starting to wonder if I am looking at the wrong tables? Can you advise?

    http://www.ipsos-mori.com/Assets/Docs/Scotland/ipsos-mori-scotland-monitor-tables-february-2014.pdf

    On page 2 the table says "Party support Scottish Parliament Holyrood First Vote Constituency" Does this mean it is weighted to Holyrood? If so then the weighted SNP vote appears to add up to 36% (302/845). We all know the actual, real life vote was 45.4% http://www.bbc.co.uk/news/special/election2011/overview/html/scotland.stm

    The sample also seems to be less than representative of the general population when it comes to national identity. The chart of page 4 of the tables says that a total of 48% of the sample feel either Scottish only or more Scottish than British. Yet the most recent census data tells us that 62% of the population feel Scottish only. Conversely, 48% of the sample feel more British or British only, whereas the census data gives that figure as around 34%.

    Finally, the sample and census data depart again when it comes to place of birth: On page 5 of the chart it says that 78% of the sample were born in Scotland. The census says the figure is 83%.

    Therefore, it appears that the sample in this poll is far more "British" than you would find in the usual population and way less representative in terms of SNP voters.

    What is your analysis, James? Scottish Skier seemed to think that the result, when adjusted, should be more like Y 37, N 43 and DK 20, but the explanation wasn't very detailed.

    Census data from here : http://www.scotlandscensus.gov.uk/news/census-2011-detailed-characteristics-ethnicity-identity-language-and-religion-scotland-%E2%80%93

    ReplyDelete
  6. Duncan HotdogstallMarch 4, 2014 at 10:56 AM

    Hi James,

    A glance through the data tables is setting off some alarm bells. I'm starting to wonder if I am looking at the wrong tables? Can you advise?

    http://www.ipsos-mori.com/Assets/Docs/Scotland/ipsos-mori-scotland-monitor-tables-february-2014.pdf

    On page 2 the table says "Party support Scottish Parliament Holyrood First Vote Constituency" Does this mean it is weighted to Holyrood? If so then the weighted SNP vote appears to add up to 36% (302/845). We all know the actual, real life vote was 45.4% http://www.bbc.co.uk/news/special/election2011/overview/html/scotland.stm

    The sample also seems to be less than representative of the general population when it comes to national identity. The chart of page 4 of the tables says that a total of 48% of the sample feel either Scottish only or more Scottish than British. Yet the most recent census data tells us that 62% of the population feel Scottish only. Conversely, 48% of the sample feel more British or British only, whereas the census data gives that figure as around 34%.

    Finally, the sample and census data depart again when it comes to place of birth: On page 5 of the chart it says that 78% of the sample were born in Scotland. The census says the figure is 83%.

    Therefore, it appears that the sample in this poll is far more "British" than you would find in the usual population and way less representative in terms of SNP voters.

    What is your analysis, James? Scottish Skier seemed to think that the result, when adjusted, should be more like Y 37, N 43 and DK 20, but the explanation wasn't very detailed.

    Census data from here : http://www.scotlandscensus.gov.uk/news/census-2011-detailed-characteristics-ethnicity-identity-language-and-religion-scotland-%E2%80%93

    ReplyDelete
  7. The Holyrood figures are current voting intention, not recalled 2011 vote.

    It's not really possible to make a direct comparison with the national identity figures in the census, because the census asked the question in a different way. But even comparing to the most recent SSAS, Ipsos-Mori's Scottish figures are a bit too low.

    As Oldnat pointed out the other week, the problem with making a comparison with the census figures on place of birth is that the census included children. I've tried to find the figures for 16+ only, but without any luck so far.

    I think what Scottish Skier does is take an average of the SSAS national identity figures over the last fifteen years, and adjust according to that. I'm not sure that's wise, because there's fairly clear evidence that Scottish identity has fallen back on the SSAS in recent years. So he's probably making too radical an adjustment.

    ReplyDelete
  8. Duncan HotdogstallMarch 4, 2014 at 12:15 PM

    Thanks James. Although I still can't believe that poll, no one in the real world is experiencing what it claims. I personally know 5 people who have moved to yes in the last month (including a tory, oh yes, he made a right song and dance about it on facebook) but have yet to meet a single person in the world ever who has moved from yes to no.

    Sorry for the double post earlier, don't know what happened.

    ReplyDelete
  9. OK, I've done a rough calculation to reweight the Ipsos-Mori figures in line with last year's SSAS national identity data, and it comes out as Yes 32%, No 51%. That isn't directly comparable to the published headline figures because it isn't filtered by certainty to vote, but essentially it has reduced the No lead by about 7%. Not quite as dramatic as what Scottish Skier came up with, but certainly an indication that Ipsos-Mori should be taken with a pinch of salt.

    ReplyDelete