Tuesday, March 25, 2014

New TNS-BMRB referendum poll puts the pro-independence campaign at 40%

A new TNS-BMRB poll on the referendum has just been released, and in complete contrast to ICM, Panelbase and Survation it shows a very slight increase in the No lead. The difference with the other pollsters almost certainly means that normal sampling variation is at play, and that the lead has not in fact increased since last time. (The figures don't even preclude the possibility that the lead may actually have fallen further - that would be well within the margin of error.)

Should Scotland be an independent country?

Yes 28% (-1)
No 42% (n/c)

With Don't Knows excluded (by far the most meaningful measure), it works out as -

Yes 40% (-1)
No 60% (+1)

So the pro-independence campaign have retained the vast bulk of the impressive gains they've made with TNS over recent months - with the possible exception of ICM, this is the pollster that has shown the greatest and most sustained progress for Yes. Just a few short months ago, the No campaign enjoyed a lead of 22% on the headline figures - that has been slashed to 14%, and the fact that several successive polls from the firm have shown a lead of either 13 or 14 points confirms beyond any doubt that the shift in opinion is real.

Blair McDougall seems to be particularly "excited" (if that's the word) by the figures for respondents who are certain to vote. Now, it's true that they do show a bigger increase in the No lead than on the headline figures, but the problem for Blair is that last month's numbers were absolutely ghastly for his side - with Don't Knows excluded they put Yes on 45%, right up there with the Yes-friendly online pollsters. That a face-to-face polling organisation could have produced such figures must have been a matter of huge concern for the No campaign, and the damage hasn't been entirely undone. We'll have to wait for the datasets to be sure, but a rough calculation suggests that Yes are on 41% among certain voters in this poll, one point higher than in the sample at large. In any case, TNS have noted themselves that the figures for certain voters are much more volatile - in early January, the No lead dropped from 15 points to 10, in late January it recovered to 17, in February it slumped to just 9, and now it has partly recovered to 14. The bigger picture is that it remains lower than in all but two of the previous seven TNS polls that have been published since the firm started producing monthly figures - in September it was a full 8% higher than now, very much in line with the trend on the headline numbers.

The biggest caveat of all about this poll is that it's actually quite a bit out of date. The fieldwork mostly predates the Survation and Panelbase polls, and it entirely predates the ICM poll. So not only is it possible that the further drop in the No lead detected by those three pollsters hasn't shown up in TNS due to normal sampling variation, it's also possible that it wasn't picked up because it occurred after some of the fieldwork took place.

I said a few days ago that TNS-BMRB somehow seem more important than other pollsters, and I'd stand by that. If it wasn't for them, we'd have a distinct lack of variety in the methodology used to measure public opinion - no fewer than five of the other six BPC pollsters who have been active in this campaign conduct their fieldwork via the internet among volunteer panels, which perhaps wouldn't be quite so troubling if the sole telephone pollster (Ipsos-Mori) wasn't producing figures that are totally out of line with the online firms. To muddy the waters even further, though, Ipsos-Mori's methodology isn't unusual simply because of the telephone fieldwork - there's also the fact that they don't weight their numbers by recalled Holyrood vote, or indeed by any sort of recalled vote. Suspicions have been raised that they may be interviewing only by landline, which they probably wouldn't do for GB-wide telephone polls (although in truth they've been incredibly secretive and nobody seems to know for sure). So it's not entirely clear which aspect of their methodology is most responsible for producing the disparity. That's why the face-to-face approach of TNS is so invaluable - it helps us to form a judgement about whether breaking out of the confines of volunteer online panels is in itself bound to produce the much bigger No lead that Ipsos-Mori are reporting.

And the verdict? Well, that depends on whether you're a glass-half-full or glass-half-empty sort of person, because if you exclude Don't Knows the Yes vote being reported by TNS in this poll is pretty much exactly equidistant between Ipsos-Mori on the one hand, and the trio of Yes-friendly online pollsters on the other. That's the case regardless of whether you look at respondents who are certain to vote or the entire sample.

Having said that, TNS use a fairly eccentric weighting procedure themselves. They do weight by recalled 2011 vote, but they dilute the benefit of that by drastically upweighting respondents who either didn't vote in 2011 or don't remember how they voted. People who do recall how they voted count for just 50% of the sample - a much lower figure than online pollsters use, and one that arguably assumes a 100% turnout in September. The net effect seems to be an artificial lowering of the Yes vote that TNS produce. The datasets from last month's poll suggest that there is a significantly bigger No lead among 2011 non-voters and 'can't remembers' than among the rest of the sample, meaning that if those people weren't being upweighted to quite such an extent the overall No lead would probably be a bit smaller. In other words, the fact that TNS are showing a Yes vote that is several points lower than ICM or Panelbase can't be put down exclusively to their non-online status - there's at least one other very obvious contributory factor.

What that says by extension about the Ipsos-Mori enigma is anyone's guess. Perhaps the fog will start to clear if a second telephone pollster (most likely ComRes) enters the fray at some point.

* * *


For obvious reasons there's not much change from the most recent update of the Poll of Polls, which showed the Yes vote at a record-breaking high. The No lead has bounced back by a mere 0.1%, meaning that it is still the second-lowest lead to be recorded to date.

MEAN AVERAGE (not excluding Don't Knows) :

Yes 35.3% (-0.1)
No 48.3% (n/c)

MEAN AVERAGE (excluding Don't Knows) :

Yes 42.2% (-0.1)
No 57.8% (+0.1)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 42.0% (n/c)
No 58.0% (n/c)

(The Poll of Polls is based on a rolling average of the most recent poll from each of the pollsters that have been active in the referendum campaign, and that adhere to British Polling Council rules. At present, there are seven - YouGov, TNS-BMRB, Angus Reid, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample. Changes in the Poll of Polls are generally glacial in nature due to the fact that only a small portion of the sample is updated each time.)

And here is the long-term trend...

The No campaign's lead in the Poll of Polls headline figures :

Sep 2013 - 20.2%
Sep 2013 - 20.0%
Sep 2013 - 18.4%
Oct 2013 - 17.9%
Oct 2013 - 17.5%
Oct 2013 - 17.4%
Nov 2013 - 17.5%
Dec 2013 - 17.1%
Dec 2013 - 16.3%
Dec 2013 - 16.2%
Dec 2013 - 15.8%
Jan 2014 - 14.2%
Jan 2014 - 14.8%
Feb 2014 - 14.8%
Feb 2014 - 14.7%
Feb 2014 - 15.1%
Feb 2014 - 13.6%
Feb 2014 - 14.0%
Mar 2014 - 14.0%
Mar 2014 - 14.3%
Mar 2014 - 14.3%
Mar 2014 - 13.6%
Mar 2014 - 12.9%
Mar 2014 - 13.0%


  1. I did suspect last month that the "Certain to vote" figures could have been a bit of an outlier and it looks like I was right. Febuary's poll had quite a lower no vote among over 55s than normal, and since over 55s are most likely to vote that explains the narrower gap last time.

    We've now had four TNS polls this year, each one with a YES vote of 28%/29% and all with a 42% NO.

    Now that every pollster has reported back after the currency speech, we can confirm that it had little or no impact.

  2. It remains to be seen whether it was an outlier, though - it was very similar to the figures from two polls previous to that. It may be that the lead is going to bounce up and down on that measure, due to the greater margin of error after the certainty to vote filter is applied.

  3. Taken from What Scotland Thinks:

    "Moreover, although TNS BMRB report that the proportion of people who think that ‘currency’ is the most important issue to them in deciding how to vote has increased from 2% in January to 5% now, it is still only no. 8 in the poll’s list of most popular answers. Undecided voters show no greater interest in the issue than anyone else."

    Fear Febuary doesn't seem to have worked.

  4. I would definitely be nervous if I was a No voter, but they are still on track to win. :(

  5. Now that we have the tables, I notice that the group that the drop in YES comes completely from SNP voters.

    The swing to NO among people Certain/Very Likely To Vote has happened completely among people who "can't remember" how they voted in 2011.

    I don't agree with them counting "Can't Remember" with "Did Not Vote" to match the 2011 abstention rate. The two groups have completely different voting intentions and likelihood to vote figures.

  6. It seems to me ALL the polls are a bit iffy.

    The online ones because it is clear from online data that YES is much more active and effective there. In addition, YES supporters seem to be more involved in the issues. So could they be over-represented in such Inet polls' lists? And can weighting really correct for that?

    Landline polls have to be dodgy. Nowadays due to spam calls not many people are willing to anwer home phones except to dedicated numbers. I wonder how many 'no shows' they get before they actually find someone who answers the phone (or opens the door) and is willing to take part?

    Finally straw polls are probably over-estimating YES due to the fact that NOs are under represented at meetings, although some of that could be 'weighted' using before
    and after votes.

    I'd like to see your views on those
    points James.

  7. I see this rather late poll has skewed your poll of poll figures. Would I take that the last two figures should be swapped round?

  8. Marcia : No, we'd have ended up at the same destination anyway. The only difference is that the tiny increase in the No lead would have happened at an earlier stage, followed by the significant drop.

    James : Yes, straw polls are completely useless. The only value of the debate polls is the 'before and after' element - it demonstrates that public opinion is not set in stone, and often it's the Yes arguments that prove more potent in winning people over.

    We don't know whether Ipsos-Mori referendum polls are landline-only, although to the best of my knowledge they've failed to clarify the situation so we have to assume it's a possibility. If they are landline-only, then in a sense they suffer from a similar problem to online polls - certain sections of the population are much less likely to be interviewed, because not everyone has a landline, and those who don't are disproportionately found in certain demographic groups. Just like online polls, there would be weighting to correct for that problem, but just like online polls there's bound to be the suspicion that weighting can't entirely eliminate any bias in the sample.

    Additionally, all telephone polls (including those that are not landline-only) suffer from the problem that people are far less likely to answer the phone than they used to be. Again, it's impossible to be sure that the people who do answer the phone are representative of the whole population (even after demographic weighting), and that introduces an element of uncertainty that wasn't there 20 years ago.

    My guess is that there hasn't been quite such a drop-off in the number of people who will answer a knock on their front door, so TNS may be a bit more reliable in that sense.