Wednesday, August 13, 2014

Yes vote soars to yet another record high among definite voters in new poll from traditionally No-friendly firm TNS-BMRB

It's been a bruising few days for Blair McDougall and the other Abominable No-Men, as their hopes of a post-debate polling bounce have receded.  We now have a second poll that was partly conducted after the debate, and the pattern is uncannily similar to the first one - the No lead has actually fallen among definite voters.  And in this case, it's fallen to its lowest level of the campaign so far in any poll from the traditionally No-friendly firm TNS-BMRB.

Should Scotland be an independent country? (Definite voters only)

Yes 38% (+1)
No 46% (n/c)

Although that's only a 1% increase in the Yes vote share, it's worth remembering that the 37% vote recorded in last month's poll was 2% higher than it had been in any previous TNS poll - or to put it another way, tonight's poll has the Yes vote 3% higher than in any TNS poll prior to July.  A rough calculation suggests that with Don't Knows excluded the position is likely to be Yes 45% (n/c), No 55% (n/c), although there's an outside chance that it might be Yes 46% (+1), No 54% (-1), which would also be a new record.  We'll find out when the datasets are published, presumably tomorrow.

On the headline figures that cover the whole sample regardless of likelihood to vote (they even include people who say they are absolutely certain not to vote!) there has been a predictable reversion to the mean since the last poll which saw the No lead slump to a record low by a huge margin.  The bad news for the No campaign is that while tonight's figures may not be the worst for them in a TNS poll, they're the second worst.  Last month's poll was the first time in the campaign that Yes had been higher than 41% in a TNS poll after Don't Knows were stripped out, and tonight is the second.  That makes it very difficult to argue that Yes haven't made further progress since the spring, unless there have been some highly coincidental margin of error effects in both of the last two polls.

Should Scotland be an independent country? (Whole sample with Don't Knows excluded)

Yes 42% (-2)
No 58% (+2)

With Don't Knows taken into account, Yes remain on 32%, which is their record high for the campaign - prior to last month's poll they had never been higher than 30% (or at least not since TNS introduced an enormous methodological change).

When the datasets are released, the first calculation I'll be making is voting intentions with the small number of definite non-voters stripped out - in both of the last two TNS polls that in itself was sufficient to increase the Yes vote by the best part of 1%.  And that gives a clue as to why the figures for definite voters only (which TNS are increasingly giving parity of esteem to) may be the more meaningful ones - they equate to a turnout in the low 70s, which is perfectly plausible.  If you think that the turnout is more likely to be around the 80% mark (or a bit higher), then the relevant figures will be for those who say they are very likely or certain to vote, and we won't find out what those are until tomorrow - although it's reasonable to assume that they'll be more favourable for Yes than the numbers from the whole sample.

As I always point out, good results for Yes in polls from TNS and Ipsos-Mori are worth more than good results from other pollsters, because those two are the only firms that don't rely on volunteer online panels for their referendum polling, and that actually seek out a sample in the 'real world'.  It's hugely encouraging that both are now showing Yes at a record high for the campaign among definite voters (42.5% in the case of Ipsos-Mori, 45% or 46% in the case of TNS) as we enter the crucial last few weeks.

And what does the TNS poll tell us about the post-debate polling landscape more generally?  It slightly increases the likelihood that the Survation poll showing an unusually high No lead was an outlier caused by random sampling variation - but the emphasis is on the word 'slightly', because what we really need to be sure is another poll that was entirely conducted after the debate, rather than only partly.

Final thought - as far as we know, TNS haven't yet joined the new orthodoxy of weighting by country of birth.  That may not be such a huge issue for them, because a face-to-face pollster ought to find it easier than an online pollster to put together a representative sample - but it would still be a very wise idea to weight by country of birth to be on the safe side, because it's such a strong predictor of referendum vote.

* * *

SCOT GOES POP POLL OF POLLS

To state the bleedin' obvious, the only reason this update of the Poll of Polls shows a slight increase in the No lead is that TNS are represented in the sample by their headline figures - if they were represented by the numbers for definite voters (as is the case for Ipsos-Mori), the No lead would have fallen back again.

MEAN AVERAGE (excluding Don't Knows) :

Yes 42.4% (-0.4)
No 57.6% (+0.4)

MEAN AVERAGE (not excluding Don't Knows) :

Yes 36.5% (n/c)
No 49.5% (+0.7)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 42.5% (-0.3)
No 57.5% (+0.3)


(The Poll of Polls is based on a rolling average of the most recent poll from each of the pollsters that have been active in the referendum campaign since September 2013, and that adhere to British Polling Council rules. At present, there are six - YouGov, TNS-BMRB, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample. Changes in the Poll of Polls are generally glacial in nature due to the fact that only a small portion of the sample is updated each time.)

58 comments:

  1. +1 is soaring? I swear you do this to troll ;)

    ReplyDelete
  2. Confirmed BT claiming their lead increasing?

    ReplyDelete
  3. Daily record page back online:

    http://www.dailyrecord.co.uk/news/scottish-news/support-no-vote-stalling-says-4039764#.U-qgCO_RyJM.twitter

    ReplyDelete
  4. Blinking: I think that tweet is based on a comparison of the full panels of both polls, so a fair comparison would think. What would be wrong would be to compare the full sample to the certain to vote sample, as you suggest.

    ReplyDelete
  5. "How come Better Together are claiming the no vote has increased by 4%?"

    Because that's what the overall figures actually say. What James has quoted here is only for definite voters.

    Now of course the definite voter figures are interesting, but I think it's pretty reasonable to suggest we should at least be acknowledging what the overall figures say as well. Maybe that's coming in an updated version of this post as he does say there are more details to follow.

    ReplyDelete
  6. I think James's "outside chance" is a very outside one.

    My calculations are -

    Excluding undecideds (changes from the TNS poll published on 14 July in brackets)
    All respondents – Yes 42% (-2%) No 58% (+2%)

    Certain to vote – Yes 45% (+1%) No 55% (-1%) – (rounded %s are the same, but underlying figures show the shift – actually 0.7%)

    Certain to vote – 73% (-1%)

    Undecideds

    All respondents – 23% (-4%)
    Certain to vote – 16% (-2%)

    So a poll that both sides can spin their own way.

    ReplyDelete
  7. Within the campaign period certain to vote is usually used by pollsters so the 38/46 should really be the headline figure

    Two points

    1. the Herald and Record headlines were pulled because they were pro-Yes so I now believe the polls are being manipulated and being used to destroy the confidence of the Yes campaign.

    2. Better Together always get details first so are able to spin any poll as favourable to them.

    We should assume the polls are no more reliable than any other BT propaganda but how to stop confidence draining away?

    Could Yes commission a poll? Or perhaps we could produce a poll from RIC canvassing?

    ReplyDelete
  8. Wise words from oldnat but not really on message.

    ReplyDelete
  9. "Wise words from oldnat but not really on message."

    Every Cabinet needs its Ken Clarke - as David Cameron will now discover.

    ReplyDelete
  10. I don't want an indy Scotland so that I can "stay on message"! I want it so that I can remain "off fucking message!"

    ReplyDelete
  11. "And what does the TNS poll tell us about the post-debate polling landscape more generally?"

    I'd be careful reading too much into that, given that the survey was "largely conducted during the games", prior to the debate.

    ReplyDelete
  12. As far as I can see it carried on for two days after the debate, which is non-trivial.

    ReplyDelete
  13. James: I was going by the statement in the Daily Record article. Is there any further information on fieldwork dates? I'd agree 'some' is non-trivial, but it may not be significant, say ~10-20%.

    "The survey was largely conducted during the Commonwealth Games in Glasgow, with some interviewing taking place after the August 5 televised debate between First Minister Alex Salmond and Better Together leader Alistair Darling."

    ReplyDelete
  14. another excellent poll

    5 weeks to go and a very healthy lead

    no commonwealth games bounce which was Alex's plan and we are still to see the effects of Alex's pandas and aliens driving on the right hand side of the road

    all very very encouraging

    ReplyDelete
  15. Another anonymous out of touch twit who simply doesn't understand that labour were just as confident and just as complacent in 2011 when the polls utterly failed to reflect the reality on the ground

    no post-debate bounce for Darling and the unionist media's demented shrieking about currency, and we are still to see the effects when Darling gets asked why he supported the Iraq War and scaremongered about Weapons of Mass Destruction.

    all very encouraging.

    Even more encouraging are the mass canvass results for Yes. Happily the BritNat twits who are too complacent and ignorant to care about GOTV and the ground campaign

    ReplyDelete
  16. The last poll in 2011 that had Labour in the lead was 27 march.

    ReplyDelete
  17. "The last poll in 2011 that had Labour in the lead was 27 march."

    Yeah, but we don't need a result of Y+18%, +1 vote will be good enough. ;-)

    ReplyDelete
  18. This referendum has convinced me that polling is simply another propaganda tool.

    ReplyDelete
  19. It's all gone Salmond -shaped for the Yesmen.

    ReplyDelete
  20. David Halliday - I believe so, which is why eg. MORI use "certain to vote" as their headline number.

    There are lots of links to actual datasets in the tables on this page:

    http://en.wikipedia.org/wiki/Opinion_polling_in_the_Scottish_Parliament_election,_2011

    ReplyDelete
  21. Brian:

    Thanks but I think if you actually follow the links you'll see overwhelmingly most of them are dead.

    ReplyDelete
  22. Panelbase poll in the field according to my pal

    Might get it for this weekend

    ReplyDelete
  23. So yes have lost then.

    If you're willing to take respondents' word about how definite they are to vote at face value, you also have to take their word about which way they're going to vote at face value.

    ReplyDelete
  24. This despite their vague anc shifting weightings, both before and after the poll.

    ReplyDelete
  25. I see yes vs No amongst men is almost back to 2011 levels when Yes went ahead (including in TNS). 48% Yes for men now.

    Ladies still being coy. This of course was the 2011 pattern. Men 'moved' first to close the gap then women shifted to hint at the landslide coming. Of course the shift actually occurred 2007-9 then hid for 2 years meaning it was more men became honest (again) first, then women.

    Yes has 84% Certain to vote, but No only 74%. So that's 36% saying they are certain to vote and planning No for now. That's quite a difference - why the reluctance among No to actually vote No?

    Unfortunately TNS haven't asked the question 'Will you yet change your mind'. That of course gives <30% solid No as per ICM's 'completely against independence' numbers and agrees with No in canvassing.

    No better hope that women, for the first time in the political history of Scotland, don't end up largely voting the same way as men. We know that women are less firmly against indy than men (ICM 1-10) and distrust Westminster as much or more. They are also saying they are more Scottish / less British (SSAS). I wonder what they'll do. If they do the same as 2011, then Yes will in all probability take it. It's pretty much that simple.

    ReplyDelete
  26. 48% ex DK for men I mean.

    ReplyDelete
  27. YouGov IndyRef poll in the field this morning. Very many questions about Salmond after a NO vote

    ReplyDelete
  28. I note also that for the first time in TNS, Yes is higher in the unweighed base than it is in the weighted base. The opposite for No.

    James - I believe you've noted this occurring in other polls? I wonder what's causing it if it is a new phenomena.

    We can see that TNS struggle much more to get people under 54 to talk to them; the group most supportive of Yes. I say struggle to get them to talk as TNS is not self selecting; people need to actively agree to take part once they open their door and are asked if they want to.

    The data suggest those age groups most likely to vote Yes are the age groups most likely not to partake in the poll - men in particular.

    ReplyDelete
  29. Well, this is fascinating.

    Large groups of people are not answering TNS polls it seems and extent of this varies with time in response to events. The groups not answering are those most supportive of independence and the SNP; younger generations.

    How can you tell? Well, simply from the total up-weighing needed for each age group. The more a group needs up-weighting relatively, the more they must not be responding.

    If there was a continuous problem reaching a particular group you'd not see any trend with time as the up-weighting needed would always be fairly consistent or at least bounding up and down a bit with no obvious trend with time.

    In 2009 when the SNP were hitting >40%, very little up-weighting was needed. No referendum on the cards and the heat on Scotland wasn't too bad.

    In 2010 election approached time (heat turned on) when Labour were apparently ahead, the up-weighting needed hits a huge spike; large numbers of people were refusing to give TNS their views - presumably SNP voters.

    This then plunges as we head into 2011 and the polls start to show the SNP gaining. It's not maybe simply lying (this did happen), but a large component lying by not answering at all. When these people started talking, they were SNP...

    In 2011, the need for weighting dropped massively (but not to 2009 levels) and of course Yes went ahead as a result; more Yes people were answering.

    Since then, the up-weighting has climbed back and is reaching record highs, which strongly suggests more and more people are shutting the door on TNS and these people are Yes. The fact that Yes is rising and No fell even with that is quite something.

    ReplyDelete
  30. Yup, the higher the total up-weighting needed to correct for those refusing to answer the poll, the lower the Yes vote.

    You can even see the 2012-13 No peaks - upweighting peaks again here then starts to fall gently in line with Yes recovering.



    ReplyDelete
  31. Yup, the higher the total up-weighting needed to correct for those refusing to answer the poll, the lower the Yes vote.

    You can even see the 2012-13 No peaks - upweighting peaks again here then starts to fall gently in line with Yes recovering.



    ReplyDelete
  32. Why do the trolls mix up what a poll company predicts with what is happening in reality? They are at best blindly trying to predict where the light switch is in a darkened stadium. At worst they are using their psephology to tamper with the public's perceptions.

    James Kelly highlights the direction of the trends in their figure and bases his views on l'actualite that the people are moving towards 'Yes'.

    ReplyDelete
  33. Thanks SS, I now understand the way polling works LOL

    What an absolute crock of shit it all is.

    'upweighting' people that vote, of course you are going to get the sorts of figures no get...as that is the only figures they can go on.

    People pay money for this?

    ReplyDelete
  34. Thanks SS, I now understand the way polling works LOL

    We might also ask ourselves why MORI struggled to get men (more Yes) to answer the phone too, with this problem peaking when No peaked / yes toughed. As the number of men agreeing to be polled has climbed, so Yes has risen again and No has fallen.

    The same age issues are also evident in addition to men; the more MORI have managed to get a representative sample (i.e. they've not been struggling with some groups, notable younger people and also men), the higher the Yes is.

    This does strongly suggest pollsters do have a problem with getting Yes people to answer polls.

    That might annoy us polling geeks, but in the end it's them voting Yes on the day that's important.

    Anyway, thank the no-campaign for demonising Yes with all the 'anti-english xenphobic racist' stuff. You want to get people to say no to answering indy polls, that's how you do it.

    I'm going to look further into it, but the TNS data set is extensive, covering many years and quite stunning. It is also non-self-selecting which is ideal for getting an idea of abstention.

    People who choose for example not to sign up to online panels is likely to be much harder to detect. I'm going to have a look though.

    ReplyDelete
  35. "We might also ask ourselves why MORI struggled to get men (more Yes) to answer the phone too"

    We might indeed since I personally was contacted by MORI three over a period of many months that clearly covered multiple polls. Each time they were only interested in the 16-24 cohort.

    The last time the Mori employee also promised to include me in a future poll after I had pointed out to them they had contacted me three times previously and they weren't interested.

    Still waiting to hear from them curiously enough and that last contact was over a month ago.

    "This does strongly suggest pollsters do have a problem with getting Yes people to answer polls."

    And that at least some of those pollsters don't seem capable of conducting those polls particularly competently.

    ReplyDelete
  36. Here's what I dont understand about all this shy Yes stuff.

    It seems the whole site agrees, even No contributors agree, that Yes are more outgoing , more excited, more anxious to campaign, keener to put Yes in their window, to celebrate loudly on social media when a single NO sinner is converted, fo knock on doors and are generaly desperate to publicise their views and quite publicly find the whole campaign a tonic for the soul.

    Moreover there is indeed hard polling evidence that Yes are happy with the length of the campaign and are enjoying the campaign while No voters are doing their best to ignore it and wish it had been over some time ago.

    Yet we are aked to believevthat a large portion of these same keen-to-participate keen-to convert Yes voters who put all these Yes posters in their windows apparenty bury their head, hide their views and/or run a mile at any prospect of airing their views iIN CONFIDENCE to a pollster? Meanwhile the bored disillusioned No voters who want it all over are apparently much more ready to give time and energy to pollsters.

    More likely surely that the bored frustrated No voters, who are not enjoying the campaign one bit, would be the ones to bypass spending time airing ther views.

    It doesnt stack up.

    ReplyDelete
  37. Yet we are aked to believevthat a large portion of these same keen-to-participate keen-to convert Yes voters who put all these Yes posters in their windows apparenty bury their head, hide their views and/or run a mile at any prospect of airing their views iIN CONFIDENCE to a pollster?

    Oh yes it does stack up - because these are NOT the same voters. The voters flying the flags are not at all shy, but there are many more keeping their heads down.

    ReplyDelete
  38. I'm a man. When i'm watching the TV and the phone rings, i pick it up.
    If it is computer generated call, or someone on the other end says ' Hi, i'm.....', i put the phone down.

    ReplyDelete
  39. Why would anyone welcome market research phone calls? I categorise these as nuisance calls and got them blocked years ago.

    ReplyDelete
  40. Expat: I have two yes stickers on my car and one in my house window, I have twibbons on facebook and twitter. I post indy comments on both daily. I wear my badge every day.

    My sister who is every bit as yes as I am won't even click the like button on the Yes Scotland page on facebook in case some no voting friend or workmate sees it and unfriends her.

    It's just how it is.

    Maybe she's just more insecure than I am, haha.

    ReplyDelete
  41. It doesn't stack up.

    Maybe it doesn't.

    I'm just reporting what I see in data and analysis of data looking for correlations is my day job (albeit production chemistry rather than social sciences).

    There does seem to be a strong correlation between being able to more easily achieve a sample without having to heavily up-weight certain groups and a higher Yes.

    This pattern also seems to following polling trends in response to events.

    I've speculated as to the cause.

    Here's a list of things I've been called for saying openly to people I support indy:

    Racist
    Anti-English
    Supportive of a dictator
    Xenophobe
    Vile
    Abusive
    Nasty
    Stupid jock
    Economically illiterate
    Living in a fantasy braveheart world

    etc.

    And I've not been called these by a few trolls on the internet, but by prominent politicians and media figures. Live on TV, in the papers etc. Even in Parliament.

    I'm made of quite stern stuff though and can argue my case. However, even then I've been shy Yes, particularly in the beginning before I felt I could fight anything thrown at me with reasoned argument. Nice Mrs McPhee from bathgate is likely less so.

    It is a well known fact that people lie in polls, even if only to avoid embarrassment about minor things. People say they voted when they didn't, they say they plan to vote when chances are they won't. They say they backed the winner when they didn't. They say they voted Labour when they, out of self interest, voted for the Tories.

    If you ask them if they ever drove under the influence, chances are they'll say no even if they did. On sensitive issues which they can be attacked for they are particularly reticent to the point they may decline to be asked at all. Breaking up Britain is about the most sensitive political issue you can get.

    ReplyDelete
  42. "Oh yes it does stack up - because these are NOT the same voters. The voters flying the flags are not at all shy, but there are many more keeping their heads down."

    That's true, of course, but the argument seems to rest on the idea that the social environment makes Yes voters more likely to keep their head below the parapet than stick their head up and risk a negative reaction. But if Yes has more vocal campaigners you would think it should be the opposite - that there are a greater number of people ready to criticise No voters which therefore makes the social environment less amenable to those expressing No sentiments.

    Anecdotally if someone pointed a gun at my head and forced me to make a judgement call on it I'd say the reaction to No supporters seems to be harsher than for those on the Yes side.

    ReplyDelete
  43. That's true, of course, but the argument seems to rest on the idea that the social environment makes Yes voters more likely to keep their head below the parapet than stick their head up and risk a negative reaction.

    No it's not the social environment we're talking about here; many people in relaxed social environments are indicating a shift to YES. It's the reactions to authority that are pertinent, because that authority constantly vilifies the whole concept of independence. Anything that has an "official" status becomes suspect.

    ReplyDelete
  44. If the No supporters on here seriously don't understand the difference between political activists/grass roots of parties and run of the mill voters, (who self-evidently hugely outnumber activists/grass roots for any party) then you have to wonder if every election and referendum baffles them.

    They are not even close to being the same thing.

    A grass roots campaign is also a very different beast indeed to the 24/7 angry shrieking from the westminster bubble media and some 37 daily/national papers in scotland who have made no attempt whatsoever to disguise their attacks on the Yes campaign and Yes supporters.

    On of the biggest points about a grass roots campaign is that it grows below the surface (the clue is in the name) and completely bypasses the overt and judgemental media landscape. it does this far more subtly by using things like word of mouth and meetings up and down scotland that are being packed out. Those meetings are where undecideds can sit in peace listening to the arguments and know they will not be bothered by a rabid and shrieking media. That intense incessant hostility from the media will obviously make voters keep their cards close to their chest.

    That was the choice of the No campaign who the decided on Project Fear and scaremongering. Let's not forget it was also the choice of SLAB when they had a negative and hostile campaign in 2011 AND 2007. (with some of the precise same idiots running both SLAB campaigns and now involved in Project Fear)

    Turns out that relentless negativity does indeed succeed in creating Shy voters who are on the receiving end of it. Those results have already proven it.

    Why on earth should that be a surprise to anyone since the pollsters STILL use corrections they had to introduce for Shy voters from as long ago as 1992.

    ReplyDelete
  45. another excellent poll for the no camp tomorrow

    survation\daily record

    1000 women polled

    34% yes

    50% no

    16% undecided

    Good stuff

    ReplyDelete
  46. Is that not a very low no vote among women?

    ReplyDelete
  47. SLAB trotted out the same seemingly low polling scores for women voters in 2011.

    Then there was the SNP landslide which showed just how much women were being affected by shy votes.

    The complacency of the No campaign is wonderful to behold after they did the precise same thing twice before in 2007 and 2011.

    No are merely pushing their already laughable grass-roots and GOTV organisation ever lower and No voters will have little motivation to go out and vote if they seriously believe it's already in the bag for them.

    The Yes campaign well never reach that level of complacency. We learned the lessons from 2007 and 2011 and we've worked most of our lives to see this vote.

    ReplyDelete
  48. "Is that not a very low no vote among women?"

    The No lead has fallen by 3% among women since the last Survation poll, yes. So not such an "excellent poll for the No camp" - but you could probably have guessed that our resident No trolls wouldn't exactly be telling you the whole truth.

    ReplyDelete
  49. "Then there was the SNP landslide which showed just how much women were being affected by shy votes."

    YouGov's final polls in 2011 got the constituency result almost spot on, so shy SNP voters can't have been much of an issue. The final rounded result was SNP on 45, Labour on 32. YouGov had:

    21st of April: SNP 45 / Labour 32
    28th of April: SNP 42 / Labour 34
    4th of May: SNP 42 / Labour 35

    So the 21st poll was pretty much as accurate as a poll gets and the other two were within or just outside the usual 3% margin of error. There isn't much room for "shy SNP voters" there.

    ReplyDelete
  50. Reporting a 7% SNP lead when the actual result was a 13% SNP lead is "almost spot on"? Let's get real here. And you can't just gloss over the mind-bogglingly huge error on the list vote.

    ReplyDelete
  51. "Reporting a 7% SNP lead when the actual result was a 13% SNP lead is "almost spot on"? Let's get real here. And you can't just gloss over the mind-bogglingly huge error on the list vote."

    A few points:

    1. The margin for error refers to the percentage a party receives, not the gap between parties. If two parties have identical "real" support it's entirely possible for a legitimate poll to show them 6 points apart - yet you've just quoted an example of the gap being 6 points off as if that proves the polling was nonsense. That's not even considering the 95% confidence.

    Whichever way you look at it there's no proof that there was some "shy SNP voter" effect in those numbers. It's all entirely consistent with an accurate methodology.

    2. The constituency poll on the 21st is undeniably spot on (which is the point at which postal ballots had been sent out - i.e. 20% of the counted votes in the election).

    3. The regional vote was off, but it's hard to imagine why this "shy SNP voter" effect would exist in the regional vote but not in the constituency vote. Are people only intimidated from indicating a preference for the SNP when they're asked about their second vote in a Scottish election? That would be one of the oddest polling phenomenons we've ever seen and if you could prove it I'd advise giving up this blog and drafting a journal article on the subject.

    ReplyDelete
  52. "If two parties have identical "real" support it's entirely possible for a legitimate poll to show them 6 points apart"

    Not within a 95% confidence level it isn't - otherwise a 6% lead would be considered a statistical tie, which isn't the case. Admittedly it's a very common misconception.

    "The constituency poll on the 21st is undeniably spot on"

    Oh well, as long as YouGov are only wildly inaccurate in two polls out of three, who's to quibble?

    ReplyDelete
  53. Just to be clear...

    The 'problem getting respondents' proxy I used (for shy) showed a huge peak when Labour were 'apparently' ahead in polls in 2010 / early 2011. This then dropped sharply as 2011 approached which would suggest the respondent problem reduced markedly close to the election. That might help explain how suddenly the SNP appeared to gain, with shy vote hitting a minimum just before the election and so not having much of an impact in final polls. People started answering who did not before.

    For the TNS data there does seem to be a close correlation between the apparent inability to reach respondents from certain (more Yes) groups and a higher No. The pattern goes up and down like the polls and correlates linearly when you plot one vs the other all the way back to 2007>.

    A shy vote would affect all polls, although finding evidence for it in online poll data may be harder (I've not looked as yet, but see it in MORI telephone) as how do we know these people volunteered for panels in the first place? We would need them to have both done so and then stopped responding to requests in large numbers to the extent that remaining members of similar age/sex were insufficient to plug the gap and abnormalities started appearing.

    Shy vote, if present, would always be more obvious in a non self-selecting poll where obtaining respondents required the most effort / time. That is TNS chapping doors. Even for MORI, the ability to just keep phoning means filling gaps is easier and quicker. Of course you are filling gaps with non-shy.

    Note shy does not mean SNP alone. It could be shy 'don't vote normally' or shy Labour Yes. In the latter case, that would result in a lower Yes in your 2011 recalled vote Labour group.

    Only time will tell. If true it means Yes may do better than polls conclude right now. How much better is not possible to predict.

    The fact is when TNS (and MORI) don't have to up-weight, they get much higher Yes and that's all we know.





    ReplyDelete
  54. Note if there is a Yes, we can expect polls taken afterwards to show more people saying they voted Yes than did.

    That would be a normal response and is typical of 'historic' wins. Plenty of examples of 'I backed the winner honestly'; the 1997 GE being a classic.

    People are never fully honest with pollsters, whether it be embarrassment, something is sensitive or simply because people are complex and not every issue is black and white so asking them different ways gets different answers.

    Pollsters and polling geeks would love it if people of every view responded equally and honestly. They don't and what effect that has on final vote is very difficult to predict.

    ReplyDelete
  55. "Not within a 95% confidence level it isn't - otherwise a 6% lead would be considered a statistical tie, which isn't the case. Admittedly it's a very common misconception."

    I'm really not sure what you're trying to argue here because on the face of it that comment doesn't make any sense. A 95% confidence interval simply means it will be within 3% of the actual level of support 95% of the time (i.e. in 19 out of 20 polls you conduct). The other one poll will be outside that range.

    What we have in those polls are (with the admitted qualifier that we don't know the rounding) two polls that are within 3% of the actual total the SNP and Labour got in the election and one poll that's virtually identical to the actual total. You calling that "wildly inaccurate" or pointing at the 6 point gap as if it's proof the polling was methodologically flawed is simply incorrect. There is nothing in those results that necessitates a theory about "shy SNP voters" because they're entirely consistent with a broadly accurate polling methodology that's simply subject to the random variations you get in all polls.

    ReplyDelete
  56. "I'm really not sure what you're trying to argue here"

    If you don't understand the point I'm making, how do you feel able to tell me with such certainty that I'm incorrect? As it happens, the point I'm making is precisely the one you think I'm making - that a 6% error in the lead is outside the 95% confidence level. You think it isn't - as I said, that's a very common misconception.

    There is no doubt whatever that YouGov had their methodology wrong in 2011. They must accept that themselves (privately at least) because they've since changed it radically.

    By the way, you keep going on about "shy SNP voters" as if that's a point I've made - I haven't. I'm simply pointing out to you that you're wrong about YouGov's results being defensible.

    ReplyDelete
  57. "a 6% error in the lead is outside the 95% confidence level."

    The reason I don't understand the point you're trying to make is that your sentence above makes absolutely no sense. You seem to be suggesting that a 95% confidence interval means "95% of 3%" (and therefore that the margin of error is actually less than 3% so a gap of 6% is impermissible). I gave you the benefit of the doubt on the first comment - that you might have simply been trying to make a completely different point and it came out in the wrong way - but you've just repeated the same thing.

    The 95% confidence interval doesn't refer to the size of the margin of error. What it means is that if you run 20 polls you'll have 19 of them within 3% of the actual total for each party, while the other poll will be outside of that range. In other words, if you run 20 polls one of them is bound to be unreliable through random circumstance - i.e. 95% of polls are reliable, 5% aren't; you have "confidence" in 95% of the polls conducted.

    What it absolutely does not mean is that the 3% margin of error is "only 95% of 3%". Instead of being argumentative about it I'd suggest you take the point on board - you're running a blog about polling after all.

    ReplyDelete