Monday, July 7, 2014

A few more bits and bobs from the TNS-BMRB poll

One thing that always amuses me when I read through the TNS datasets is the crossbreak that shows how the people who are certain not to vote in the referendum are planning to vote in the referendum.  In this case, 50% of definite non-voters are "planning to vote No", 10% are "planning to vote Yes", and the remainder are "undecided".  What exactly is the point of the voting intention numbers taking account of people who have already decided not to vote?  I can only think of two realistic explanations - a) it's assumed they might not have properly understood the question about how likely they are to vote (in which case it would probably be better to improve the wording of the question), or b) it's assumed they might not be telling the truth.

For what it's worth, though, if we take those people at their word and strip them out of the voting intention numbers, it's enough to reduce the overall No lead by 1.2% at a stroke (the 58.6%-41.4% lead becomes a 58.0%-42.0% lead).

Although this shouldn't really matter, it's intuitively encouraging to see that the Yes vote in this poll is very nearly as high on the raw numbers as it is on the weighted results.  That doesn't often happen, because Yes support is concentrated in groups that typically need to be upweighted in polls (such as younger and lower-income people).  It's also a relief to see that respondents' vote recall on the raw numbers reflect the fact that the SNP won the 2011 election comfortably.  There have been so many previous TNS polls in which more people recalled voting for Labour than for the SNP that I was beginning to wonder if false memory might be mucking up the weighted results, but thankfully that's not a concern in this particular poll.

As we've discussed a number of times before, TNS have a very odd weighting scheme which essentially assumes a 100% turnout.  People who say they didn't vote in 2011 or who can't remember how they voted are sharply upweighted to match the rate of abstention in the 2011 election.  But for the life of me, I just cannot understand the logic of treating people who can't remember how they voted as if they were non-voters.  Again, the assumption must be that all of them are lying or misremembering, but even if that's the case (highly unlikely), how can TNS possibly calculate the correct target figures for the proportion of 2011 non-voters who can be assumed to now be wrongly saying that they voted but can't remember how?  It's hard not to conclude that there must be a hell of a lot of random guesswork going on here.  As it happens, people who say they can't remember how they voted in 2011 almost always break heavily for No in TNS polls, which means that if the upweighting doesn't have a rational basis, then it will be artificially boosting the No lead (albeit only by a modest amount) in the headline numbers.  In this case, the 92 'can't remembers' in the raw numbers are upweighted to count as 135 'virtual' respondents.

The reality is that a lot of people who didn't vote in 2011 are probably embarrassed to admit that to pollsters, which means that what TNS are doing is fundamentally misconceived - if they want to ensure non-voters are properly represented, they should really be looking for some of them in the "voted Labour/SNP/Tory" columns.  The mistake may not be having a huge net impact on the headline numbers, because people who openly admit to not having voted in 2011 (as opposed to the 'can't remembers') actually split between Yes and No in much the same way as the rest of the sample, meaning that upweighting them sharply may not cause a massive distortion.  But an unwise weighting scheme still introduces a degree of uncertainty into proceedings that we could well do without.


  1. The main thing I see with TNS 'chap your door' and MORI 'phone you up out of the blue' is due to the lack of anonymity, Yes support drops sharply at the expense of DK and No.

    MORI has the additional landline telephone problem of hitting older, more conservative households; landline use dying out
    in the homes of the younger, social media generation who are much more Yes orientated.

    Recent Panelbase / MORI/ TNS

    SNP 2011
    76% / 75% / 65% Y
    14% / 11% / 15% N

    29% / 11% / 17% Y
    67% / 75% / 65% N

    23% / 11% / 11% Y
    67% / 83% / 75% N

    Yes more than halves amongst Lib and Lab in TNS and MORI; that's very significant.

    In MORI, the recent Yes rise seems to be largely associated with SNP voters stopping telling porkies; support now starting to reach online levels when it didn't before, but lay between TNS and online.

    This phenomenon is most evident in long term trends.

    Both MORI and TNS showed the highest No peak / Yes trough during the 2012-2013 period where No leads of 25 points were the norm.

    This didn't happen in panelbase; in fact it's non-existent.

    We might ask ourselves why people opening the door to a stranger / answering the phone to one who asks all their details then their view as to whether they are an 'economically illiterate, English-hating, braveheart loving nazi' according to the media are much more likely to say 'No' or 'DK' than people filling out a form in a much more anonymous manner on a smart phone.

    ICM desperately looked for shy No and couldn't find any; their first attempt suggested the opposite. They then looked again and we've not seen the results...

    Telephone and face to face polls are much more prone to people giving an answer that they feel most comfortable giving / the answer that the feel would not put them on the spot.

  2. To add...

    People lie to pollsters. They do it all the time. It's a headache for pollsters as it's extremely difficult to gauge the extent of it.

    Trying googling the subject; a plethora of articles.

    You can't ask the respondents if they are lying because if they are, they'll say 'No I'm not'. So, all you are left with is anecdotal evidence such as we see for Scotland regarding the SNP in 2011 and the mirage of a Labour lead plus what I've discussed above.

    Invariably, people lie on sensitive issues where they feel their view might open them up to attack / compromise their position. The shy Tory vote in the UK is a long standing issue and why Labour's current lead is even weaker than it seems.

    The position the media portray as the norm is the one people will feel most comfortable saying. In Scotland, that's No or DK.

    Better Together should pray that Scots are not doing what they did on a mass scale ahead of 2011.

    We couldn't prove it then, but had widespread evidence for it. We can't prove similar is occurring now for Yes, but again have widespread evidence for it.

    We'll just need to wait and see.

    Given the lack of polls we are getting, we might not know the result until the votes are counted. People of course don't lie when alone in the voting booth.

  3. I enjoyed reading this article and the comments by Scottish skier add to the whole debate about polls. About a year ago I was interviewed at home at length by IpSos Mori to be included in their dataset for the referendum. I gave honest answers which were that of a slightly left wing, middle aged middle class SNP voter. I have never been contacted since!
    In my opinion the referendum pollsters are making it up as they go along to suit themselves.What you have described here, and in other articles support my thoughts.
    Keep up the good work trying to untangle it for us mere mortals.

  4. TNS and the others did not do so great in 2011 I believe the same is happening now.I am canvassing round the doors and no weighting down weighting up weighting goobledy gook same as 2011 and I and others as can be seen on social media from returns note YES in front.
    16h: To any yessers worrying about recent polls ,and to naysayers who feel emboldened some history
    It will not stop me doing everything I can between now and Sept18th ,however I believe we are already in front.

  5. I enjoyed reading this article

    I suspect that those more active in campaigns, the more politically passionate, those posting on forums etc, are far more likely to be honest to pollsters. I am of course honest too.

    However, most people are not politically active and can be very sensitive about what to them are private views. I was so pleased to discover my first 'shy' respondent in the form of a mate's mother who lied to MORI (said DK instead of Yes).

    The response rate for telephone polling is terrible. Who knows how many people decline to answer at all simply because they don't want to stick their head above the parapet. Same for door TNS knocking.

    Over the past couple of years Scotland has been under sustained bombardment for having the audacity to question it's place in Britain. It has been treated appallingly; anyone with any indy inkling is vilified in the most repugnant way, compared to nazis, a virus... It is surprising that people shut up?

    Just imagine what it would have been like if polls had continued to show Yes ahead as they did in 2011 when the SNP win briefly made it ok to be more honest. Or as they have done at various points during devolution?

  6. The worst news for the No side is the number of people who say their campaign is negative or ineffective. That is an assessment of their 'brand'. What the No campaign are banking on is that a majority of people who don't rate their brand will choose it anyway. Seems unlikely to me.

  7. who say their campaign is negative or ineffective.

    Everyone agrees on that; even the media to an extent.

    So no shyness there.

  8. I see your point ScSk. What do you think that reveals about voting intentions?

  9. The thing that concerns me as someone campaigning in an affluent area of the Borders is that the optimistic people are getting their optimism from particular local conditions. Maybe I'm just sensitive, but I hate being shouted at over a garden gate by someone calling me a racist. It makes me despondent. (On Saturday I set off to catch up with just 15 newspaper deliveries that had been missed right up at the wells of the Tweed. I found myself muttering, please don't let anyone shout at me this time!)

    I get nervous, I get discouraged. I creep up to people's letterboxes, hoping to get the paper in the door and away before I'm spotted. Encountering two houses sporting "Yes Windaes" in Tweedsmuir was practically enough to make me burst into song.

    Yesterday after church I was talking to a retired lady who is also terrified of a No vote - but who can't persuade her husband to vote Yes. That's four of them now.

    I worry that Yes may really be well behind once you factor in the affluent Tory No areas people don't talk about so much.

  10. Rolfe,

    I'm near Lauder and my MSP is SNP. Head of SBC is a Yes too.

    It's only those blue areas right at the border where Yes will struggle somewhat. 5% of the population.

  11. I'm nowhere near the border and my MSP is SNP as well. Boy have we some angry No people, and some non-angry but stubborn ones as well. But I suppose I can take comfort in the realisation that my core patch only needs 1000 newspapers and even the extra bit I took on to help a friend for the last one was only another 150 or so.

    Hell of a big area on the map, mind you.

  12. "TNS and the others did not do so great in 2011"

    TNS did pretty well in 2011 - their final poll actually overestimated the SNP's lead on the constituency vote. The real shocker in 2011 was YouGov.

  13. TNS picked things up on the 27th of March. Only then did they get the SNP narrowing the gap on Labour. So what, 5 weeks ahead and Labour still on course for a victory. Their final poll did however call the result fairly well, but that was, as James notes, just days before.

    True that Yougov were on balance the worst, even just days ahead.

  14. Yougov's failure to predict the May 2011 Scottish election is a (no doubt very comforting) myth. In the constituency vote (which like the referendum is First Past the Post and all that need concern us here) they correctly predicted the SNP would win in all their polls from 28th March to 4th May. it was close in the 4th May one but the margin generally increases. Their last really wrong poll (though it may have matched the voters mood at the time) was on 22nd February when they had SNP on 32% and Labour on 41%/ But please note a week later on the 2nd March TNS (who are lauded here) were far worse calling it 28% for the SNP and 44% for Labour. And even Panelbase, so beloved of SNP, had it on the 4th April as a dead heat after Yougov was already showing the SNP ahead! So you can't possibly single out Yougov.

    And it is not really so surprising for the polls to show voters heading back to the status quo in the last few weeks of an election. They often do. For the referendum the question will I suppose be is the status quo the SNP or the union. I guess it's the latter since that is currently how things stand and will leave the SNP still in power.

    Incidentally as en expat Scot I am utterly baffled by all this talk of people terrified to say they are voting yes. Why? Opinion polls are confidential surely. Also accusation of being unpatriotic and treacherous are just as likely to cut the other way. And how would the neighbours know anyway? Please enlighten me.

  15. Incidentally as en expat Scot I am utterly baffled by all this talk of people terrified to say they are voting yes. Why?

    Why do large numbers of people in the UK feel they need to hide the fact they plan to vote Tory? A historical nightmare for pollsters; so much so they've tried to develop methods to account for it. That's the part of government right now...

    The simple reason is a lot of people just aren't comfortable about revealing their intention on sensitive matters, and ending Britain is quite a biggie.

    If people are holding back, presumably its for similar reasons a large section of the electorate said Labour ahead of 2011 when they planned to vote SNP.

    In 2008-2009, when there wasn't the same heat on Scotland as no election was imminent, these people said 'SNP' to pollsters. Then, as the anti-SNP propaganda began, so those saying SNP reduced as they started saying Labour instead. They then went out and voted SNP.

    It wasn't an unprecedented swing at the last minute as people made up their minds. Rather, they already had some time ago largely. A few years in fact. They were looking for an excuse to say so though.

    If you are out of the country you may be unaware of what the pro-UK campaign are doing.

    They are attempting to demonise anyone who plans to vote Yes.

    Here is the Scottish Sun today even losing the rag and telling them to stop calling quiet and retiring old Mrs McPhee - who'd never consider putting a big Yes sticker on her car because it's her business and she doesn't want any bother - from Perth a Nazi.

    Now us political geeks can take such stuff. We don't represent most people though.

  16. "they correctly predicted the SNP would win in all their polls from 28th March to 4th May"

    So what? The SNP were so far ahead by the end that any pollster sill showing Labour in the lead might as well just have shut up shop completely. The issue is how far out YouGov was on the percentage for each party - and they were very inaccurate compared to most others.

    "it was close in the 4th May one but the margin generally increases"

    I don't know what that means. Other pollsters were getting it more or less right by then, so what special alibi do YouGov have?


  17. Have you seen that Ladbrokes the bookies have got a new bet out for the Referendum:

    Polling Wars
    Which company’s final poll will be closest to the result?
    YouGov 2/5
    Survation 7/4


  18. Have you seen that Survation completely demolished YouGov's absurd "Kellner Effect"?


    Scottish_Skier also previously demolished the feeble minded nonsense that one or two YouGov snapshots in 2011 meant that they weren't laughably wrong for the rest of the campaign and result.

    David McLean -'A previous poll they did about two weeks before the vote had the result almost spot on: SNP at 45, Labour at 32 (exactly what happened).

    Scottish_Skier - If you look at all their polls, this one was a big outlier. The rest, including the last two ahead of election day with SNP leads of 8 then 7 (actually 14+) show a lovely steady trend of narrowing gap, but nowhere near reality.

    If you want to include the rogue one you mention, then Yougov predicted a big Labour surge at the last minute across three polls ahead of polling day. ;-)

    Of course those of us who were there and campaigning in scotland during 2011 know perfectly well YouGov was laughably wrong.

    Have you seen that Cameron and Miliband are both on a par with Kinnock in public leader ratings ? (with calamity Clegg being even worse)


    Better Together?

    I don't think that's quite the message the westminster child abuse cover-up story dominating the papers, the News (and currently being talked about all over social media like Facebook) is conveying.

    Hard to believe the incompetent westminster establishment could seem even more untrustworthy and repulsive than after the expenses scandal, buy they've done it.

    After Coulson now we have Cameron's friend and senior aide Patrick Rock still to play out. Somewhat unfortunate timing, to say the least.

  19. Speaking of Coulson, for those who haven't already seen it..

    BBC Breaking News ‏@BBCBreaking 1h

    Andy #Coulson to face perjury charges over Tommy Sheridan trial, indicted to appear in Glasgow court on 6 August

    So much for the PB tory twits non-story".


  20. Re Yougov in 2011 James Kelly writes:

    "The issue is how far out YouGov was on the percentage for each party - and they were very inaccurate compared to most others."

    This is just plain wrong whichever way you look at it.

    The final Yougov poll on 4th May 2011 was 3.39% out on the SNP, 3.31% out on Labour, 2.91% out on Con, 0.07 out on LD. This is hardly massive. As to 'most others' Panelbase were nowhere near and the last TNS poll was .39 out on SNP, 3.89% out on Labour (ie a bigger error than any of Yougov's), 1.09% out on Con and 2.07% out on LD.

    Therefore none of this is very far outside the margin of error and, contrary to some versions, it does not remotely equate with massive polling blunders of the past. Moreover as has been noted Yougov were pretty well bang on the result two weeks earlier in their Scotland on Sunday poll and, though the IPSOS poll that Sunday was also close, that Yogov poll is actually the closest anyone got in the whole election to predicting the exact result between the SNP and Labour So much for their inaccuracy! The truth is on the 2011 election run of polls a commercial company seeking accuracy would probbaly select Yougov as the best of the bunch in a fair but not admittedly brilliant field

    Just for the the record I have nothing to do with Yougov by the way and am as anxious as anyone and as interested as anyone to hear Peter Kellner's reply to James's points but flagrant inaccuracy serves nobody.

    However I should point out that there was a mistake in my original post for which I apologise. I wrote the SNP margin over Labour 'was close in the 4th May one but the margin generally increases' and James was entirely right to say this is meaningless. I intended to say it was close in the Yougov 28th March 2011 poll, that is the first poll anyone had done since 14th February 2011 that showed the SNP in a slight lead and from there on Yougov showed the lead growing markedly. So the idea it kept on showing big Labour leads is crap. The actual progress of SNP lead over Labour in all Yougov polls from 28th March was +3, +13 +8 and finally +7.

    The only way it seems you can really castigate Yougov who emerge as accurate if not more accurate than anyone else (though obviously the fact they called the SNP short by 3.39% infuriates some) is this extraordinary idea that the SNP had been in the lead for months and months and either masses of the voters were lying or almost all the polls were all wrong. This is flying in the face of all opinion poll logic and, even if it were true, there could be no way anyone would or could ever know which is as good as pure fantasy.

  21. Anon : I saw our old friend Mike "can't be arsed" Smithson make a comment about the Survation v YouGov odds which I actually agree with. He pointed out that in this context a bet on Survation can be seen as a de facto bet on ICM, who tend to produce very similar figures, and given the respective pedigrees of the firms it would be crazy to think that ICM have less than a 50/50 chance of coming out on top in a straight contest against YouGov. The conclusion was that betting on Survation is undoubtedly the value option.

  22. "The final Yougov poll on 4th May 2011 was 3.39% out on the SNP, 3.31% out on Labour, 2.91% out on Con, 0.07 out on LD. This is hardly massive."

    It's outside the standard margin of error for both Labour and the SNP, and the problem is compounded because the challenging party is being overestimated (over and above the MOE) at the same time as the incumbent party is being underestimated (over and above the MOE). With sound methodology, YouGov shouldn't really have been underestimating the SNP lead by more than 5 points at the absolute most. That leaves only two possible conclusions to draw - either a) this was the rogue poll you'd expect 1 time in 20, or b) the methodology was misconceived. Given that YouGov's penultimate poll was very nearly as bad, I'd say the latter possibility looks the most likely.

    TNS were more accurate than YouGov - their average error was significantly lower. I've no idea why you've got a bee in your bonnet about that comparison anyway, because both YouGov and TNS are relatively No-friendly pollsters (as are Ipsos-Mori, who were also more accurate than YouGov in 2011).

    As for Panelbase, I'm struggling to work out which poll you're referring to. I have a horrible feeling you might be comparing apples with oranges by talking about a Panelbase poll that took place weeks and weeks before the final YouGov poll.

    "So the idea it kept on showing big Labour leads is crap."

    There's one slight flaw in that retort - no-one on this thread is actually saying that YouGov kept on showing big Labour leads.

    By the way, your suggested reason for believing that we can just discount YouGov's enormous error on the list vote is totally unconvincing. Margin of error is exactly the same regardless of whether you poll for a PR or FPTP election.

  23. So to sum up on 2011 the idea Yougov was miles off and everyone else was much better is just plain wrong. They were outside the standard of error on the final poll in the case of the SNP by 0.39% and on Labour by 0.31%, well within it for Con and almost completely accurate for LD. If this is your idea of a shocker you don't get out much. TNS were accurate on SNP, a whopping 1.69% outside the margin of error for Lab and just within it for the other two. The last IPSOS poll somewhat earlier (barely anyone else was polling) inflated the Labour figure and was outside the margin of error (another shock for you! ) for Con.

    Overall Yougov were predicting a healthy majority for the SNP from three weeks before the election. So sorry there really is no evidence to challenge their methodology on those figures though it is true they were polling four times as much as anyone else so any volatility is magnified. Panelbase barely polled at all except for one on 4th April showing a dead heat between SNP and Lab but please note that on either side of that (rogue?) poll Yougov were in contrast showing the SNP ahead!

    Re the list vote yeah bad even if TNS was way outside the margin for the SNP too. But irrelevant here as PR makes all polling and second choices and stuff vary hard to call. No second choices in the referendum so whatever science was applied or misapplied in that instance it won't be applied here.

    By the way I like the site and good to see people who know their stuff even if I disagree. And I hope Kellner does reply.

  24. The difference between the last Scottish election and the Indy Referendum, is that the outcome will have huge implications the Westminster Treasury.
    It will also have a profound effect on Englands reputation abroad as other nations will see 'England' Breaking up! (If you've lived overseas you will know what I mean)

    We can all see that UK gov is desperate to cling on to Scotland and have already participated in some very dishonest/deceitful behaviour.

    The question I ask myself is: would the UK gov be prepared to influence the results of polling numbers if they could find any that were corrupt enough to 'play ball'.

    Please note the UK govs behaviour in relation to the non release of Ipsos Mori results that cost them £300,000 and that they did give to the No campaign.

    Because the Yes campaign seeks to end the Union, They need to generate enthusiasm that is 'catching' and this puts polling companies into a hugely significant position.

    Although not perfect they are seen as the best source of information by most members of the public, as to how other are voting.

    If people who are not sure or who may lack the intellect to separate the wheat from the chaff, they will look at what others are doing and think that since everyone else is voting 'this way' it must be safe to do so.

    This is why the BT campaign and in particular Blair McDougal continually release statements to the effect that No is way ahead in the polls.

    So YouGov (is there a clue in the name?) are getting results that indicate Yes are way behind and falling (exactly what the BT campaign want.

    The other pollsters,show Yes on a far healthier percentages,as well as a trend for increasing support for the Yes vote, exactly what Yes wants.

    One thing we can all know that is intuitively wrong, is the YouGov ludicrous 15% English born respondents in their polls.

    Since English Born people lean heavily towards No, we can see the vote is skewered already with just this one decision.

    I will close on this, one thing people defending yougov tend to mention is that if yougov get the result badly wrong it will damage their reputation.

    I don't think this is a problem for yougov as they may simply not care about the Scottish market for polling after the referendum and instead be far more interested in the 'Rewards' of a close and friendly UK Gov who are thankful that yougov 'did its bit' for old blighty, in it's time of need.

    'Lord'? Kellner can expect a lot of government contracts for his company whatever the outcome of the referendum.

  25. Yougov last 3 polls average deviation (includes that outlier):
    -5% SNP
    +4% Lab
    -1% Con
    +1% Lib

    SNP lead over lab deviation = -9

    Yougov last 2 polls average deviation:
    -6% SNP
    +5% Lab
    -1% Con
    +1% Lib

    SNP lead over lab deviation = -11

    So, well outside variance and a huge underestimation of SNP lead over labour.

    I'm not sure why this is causing so much discussion. Yougov always overestimate Labour and underestimate SNP.

    We've even had them explain this to us; i.e. why their method increases Labour numbers and reduces SNP with motives.

    For reference.. SNP lead over lab deviation:

    TNS last poll = 0
    MORI last poll = -5

    Of course TNS are not using the same method now; the new method reduces Yes and increases DK.


    Anyway, this Westminster paedophile ring is getting more and more prominence.

    Thing is, I don't find it surprising at all.

    I imagine if you asked most Scots they'd feel similar and that it just confirms the nature of Westminster.

    Can you believe such a thing would ever happen at Holyrood?

    That's what Scots will now be asking themselves.

  26. Expat : "Overall Yougov were predicting a healthy majority for the SNP from three weeks before the election."

    That isn't true. To a large extent, the overall number of seats is decided by the list vote, and YouGov were showing a 3% SNP lead on the list - just 1% higher than in 2007! In fact I remember writing here that I was slightly worried that the SNP might yet lose the election, because in the 2003 Holyrood election YouGov had been much more accurate than other pollsters.

    Thanks for the kind words about the blog, by the way.