Monday, May 19, 2014

Concerns mount over ICM's sudden change in methodology

The datasets for ICM's referendum poll have been published, and as a result it's become clear that the methodological changes have gone way beyond the one flagged up by John Curtice and trailed by Martin Boon last month, namely the introduction of a likelihood to vote filter.  In the April poll from ICM that produced a virtual dead heat (a No lead of just 3%), the referendum question was asked first.  But in this new poll showing a bigger 12% No lead, the referendum question was asked third - which ought to set alarm bells ringing immediately, because we know that responses to later questions can be substantially influenced by the wording of earlier ones.  Indeed, John Curtice has spent a fair bit of the last few months rubbishing a poll conducted last summer by the BPC-affiliated firm Panelbase, simply on the basis that it asked the referendum question third.  That poll of course showed Yes in a 1% lead.  A few cynics might wonder why he pounced on methodological bad practice in a poll that showed movement towards Yes (and in retrospect he was probably right to do so, because it was completely out of line with other polls), but hasn't said a word about the same problem in a poll that appears to show a favourable trend for No.

It is of course necessary to look at the wording of the two questions that precede the referendum question before jumping to the conclusion that they might be distorting the result.  The second question is entirely innocuous - it simply asks how certain people are to vote, and as that's a dry, technical question there's no problem with asking it before the main question (other pollsters do the same).  But the first question is a different matter entirely.  It asks -

"Thinking about the referendum on independence for Scotland, do you feel comfortable or uncomfortable being asked which way you might vote?"

Frankly it's hard to think of a dodgier question to ask right at the start, because when respondents come to the referendum question a few seconds later they'll be thinking to themselves, "ah, this is what some people are uncomfortable about being asked", which is bound to influence their reaction in at least some cases. Indeed, a fair chunk of people will have already indicated that they ARE uncomfortable about being asked, which arguably gives them implicit permission to do one of two things - to answer 'Don't Know' even though they do know how they will vote, or to give a completely dishonest answer.

If there's been an explanation offered for the introduction of the 'comfortable' question, and for its positioning within the question sequence, I haven't been able to track it down so far. My best guess is that this constitutes ICM's 'investigation' into the possibility of 'Shy No Syndrome', which Martin Boon revealed last month was on its way. It was already a matter of huge concern that ICM distrusted their results simply because they were so favourable for Yes, and seemed to be actively looking for excuses to adjust the Yes vote downwards, rather than open-mindedly testing for any bias that might be occurring in either direction. Even so, I naively assumed that they would be able to carry out their 'investigation' in a manner that did not compromise the integrity of this month's poll, but sadly that does not seem to have been the case.

So what did this experiment discover? If the point of asking for people's comfort levels is the assumption that some of them are answering dishonestly due to social pressure, what ICM would have been looking for to prove their theory about 'Shy No Syndrome' is evidence that a greater number of people who say they are voting Yes are uncomfortable about being asked the question. In fact, it turns out that the opposite is true. No fewer than 70% of people who say they are voting Yes are "very comfortable" with being asked the question, compared to just 52% of people who say they are voting No. Just 5% of people who say they are voting Yes admit to being uncomfortable, compared to 9% of people who say they are voting No. Whether this disparity came about because ICM effectively gave people 'permission' to feel uncomfortable, and as a result a chunk of respondents who would otherwise have said Yes switched to No (thus distorting the headline referendum figures), is anyone's guess. But given that this very peculiar methodological change coincides with an increase in the No lead that two other pollsters have failed to replicate, it's an obvious suspicion to raise.

Scottish Skier suggested that I should remove the ICM poll from the Poll of Polls due to this bad practice. I can't really do that, because the whole point of the exercise is to throw all the wildly varying methodologies from BPC pollsters into a pot and see what the blend produces. And this certainly isn't the first time that a BPC pollster has used dodgy methodology - remember YouGov's embarrassingly biased preamble about Scotland "leaving the United Kingdom"? But it's undoubtedly a big concern, and it may well be distorting the trend in the Poll of Polls. The best way of judging the real trend for now is to look at recent polls conducted without a change in methodology - and TNS-BMRB, Panelbase and Survation are all showing a relatively stable position, with Yes consolidating the big gains made over the winter.

Incidentally, there are a couple of pieces of good news from the ICM datasets - the Yes vote on the unrounded figures after Don't Knows are excluded is 42.4%, which is obviously a touch higher than the rounded figures for publication suggested. And as seems to be the case every month, the Yes vote rises a touch (this time to 43.2%) after Don't Knows are asked how they are leaning.

23 comments:

  1. " If the point of asking for people's comfort levels is the assumption that some of them are answering dishonestly due to social pressure, what ICM would have been looking for to prove their theory about 'Shy No Syndrome' is evidence that a greater number of people who say they are voting Yes are uncomfortable about being asked the question. In fact, it turns out that the opposite is true. No fewer than 70% of people who say they are voting Yes are "very comfortable" with being asked the question, compared to just 52% of people who say they are voting No."

    Is that right? If we're talking about Shy No Syndrome, surely we'd expect the people voting No to be the uncomfortable ones. If you're comfortable about something, why would you be shy of admitting it?

    ReplyDelete
  2. I had the same response as RevStu - it looks like the results point to greater shyness from No voters.

    ReplyDelete
  3. "Is that right? If we're talking about Shy No Syndrome, surely we'd expect the people voting No to be the uncomfortable ones. If you're comfortable about something, why would you be shy of admitting it?"

    Yes, you'd expect people who are voting No to be the uncomfortable ones, but you'd also expect some of those people to tell the pollsters they are voting Yes. The whole point of what ICM are doing is that they're looking for people who are answering the referendum question dishonestly, and by definition therefore those people would be showing up in the 'wrong' column. If there was a Shy No Syndrome, respondents answering dishonestly due to discomfort would show up in the Yes column, and if there was a Shy Yes Syndrome, respondents answering dishonestly would show up in the No column.

    ReplyDelete
  4. That's an interesting (and creative ;)) interpretation.

    ReplyDelete
  5. ICM have really screwed up here and interpreting what went on in the minds of respondents is not easy.

    Trying to judge a shy vote is very difficult. How do you get people to admit to being shy about something! They may very likely be shy about being asked about being shy if they are shy about VI!

    Ok, the fact that the C question was asked first means trying to tie it to Y/N is questionable as it is highly likely to have affected Y/N. The big disparity from the last ICM poll suggests that's exactly what happened as the methodology hasn’t changed hugely otherwise.

    Ok, so shy No or Shy Yes voters.

    Shy No voter:

    Uncomfortable with stating intention? X (maybe, I might not admit to being shy)

    Yes or No to indy? Well, I’m not comfortable with saying No, so tick X to Yes or DK, otherwise I’m not a shy No!

    Result = Yes stable or rises, no falls.

    Shy Yes voter:

    Uncomfortable with stating intention? X (maybe, I might not admit to being shy)

    Yes or No to indy? Well, I’m not comfortable with saying Yes, so tick X to No or DK, otherwise I’m not a shy Yes!

    Result = No stable or rises, Yes falls.

    What happened? The latter.

    The fact Yes fell when ‘uncomfortable’ was associated with having a view on the indy ref much more strongly implies that Yes people felt uncomfortable and so became more shy.

    We can’t prove it, but the other way around seems far less plausible.

    The remaining Yes voters are the ones mainly not shy about Yes, hence a higher fraction of these are ‘comfortable’.

    No and DK have acquired some shy Yes, hence more respondents uncomfortable.

    There is a shy vote (the fact that it seems to be 20% for some groups in an online poll is quite significant) and the evidence suggests it is Yes.

    Otherwise, why e.g. does Yes for SNP voters sit as low as 55% for face to face polls, but is well over 70% for much more anonymous online polls? No however hardly varies.

    This strongly suggests shy Yes but No content to state so no matter how people are asked.

    Anyway, fair enough James on keeping it in your PoP but this poll must be considered an outlier and can't be compared with past ICM polls. This was very sloppy by ICM and highly questionable.

    We'll have to wait for the next one which if ICM have any sense, will not have a dodgy Q beforehand.

    ReplyDelete
  6. It's certainly not 'creative' - it's simply a fact that dishonest No voters will not appear in the No column, and ICM will know that. I don't fully understand what they think they're proving with this test, though - in order to establish a Shy No (or Yes) Syndrome, what they'd really need to do is set up a test where a sample is split in two, with one half being asked the question in conditions that match as closely as possible the anonymity and secrecy of the actual voting process.

    ReplyDelete
  7. PS. My last comment was a response to Commentor.

    ReplyDelete
  8. I see we thought on the same lines James.

    In simple terms, ICM have implied voting a particular way could be seen as bad in that it makes people uncomfortable admitting it. They let the respondents decide what's the bad way to vote which they'd be uncomfortable admitting.

    Who got hit when they did that? Yes did. That strongly implies shy Yes.

    ICM know it too, otherwise we'd have 'Shy no voters likely to swing it for the union' headlines which is what ICM wanted to be the case.

    ReplyDelete
  9. Measuring shy votes is very hard to do as people who are shy are not so likely to admit they are.

    It's hardly as if the comfortable Q would make them comfortable in admitting intention. Rather the word uncomfortable is likely to increase discomfort and encourage shy voting. This hit Yes hard as noted which strongly implies shy Yes.

    As James says, you need to make people feel as anonymous as possible to get close to the truth. Even online that's not possible (in fact people feel increasingly less anonymous here these days).

    A polling card in an envelope with just the area they live in as the only identifier would be about your best hope.

    ReplyDelete
  10. "The fact Yes fell when ‘uncomfortable’ was associated with having a view on the indy ref much more strongly implies that Yes people felt uncomfortable and so became more shy."

    Nah. The fact that No rose after the interviewee was asked about being uncomfortable is because No-voting interviewees examined their mood and worked harder to overcome their higher discomfort after being made aware of it (gave them the feeling the surveyor was sympathetic to their plight), and then they answered the indyref question more in line with their true intentions.

    So, the survey, which showed that more No voters were uncomfortable about expressing a preference, showed that more No voters were uncomfortable about expressing a preference.

    Be creative enough and you can come up with an explanation that backs up your position. And skier, I must say - you're an expert at that. You've got a place in the hearts of many a Yes voter, because you tell em what they want and need to hear.

    ReplyDelete
  11. "This was very sloppy by ICM and highly questionable."

    That's putting it mildly Scottish_Skier.

    I'm amazed at all this having just read on up on Martin Boon's 'explanation' for why they suddenly decided to push the 'shy No' narrative with such incompetent force.

    ICM actually admit to doing all this on the purely anecdotal basis that answering a No poll could be seen as unpatriotic and anti-scottish. Yet for some reason it seems to have escaped their attention that there has been a wall to wall press and media villification of Yes and SNP supporters at every turn. (Which sure as shit hasn't stopped now with the pathetic attacks on the Weirs.) Boon also has to admit that there is already polling evidence to doubt this shy NO narrative.

    "As always there’s counter-evidence to confuse things. It’s hard to reconcile that one of the two largest groups who currently “don’t know” are 2011 SNP voters. Intuitively, one might imagine this would be the last group of people who are shy to admit where they stand on Scottish independence."

    It's not that hard to reconcile if you approach it with an open mind and are not relentlessly pushing for one particular 'explanation'. Namely that there could indeed be shy Yes voters because of the media villification ICM ignores, or that quite simply the whole shy voter phenomenon has been ludicrously overblown in the first place. It certainly didn't happen in 2010 despite some pundits making a complete fool of themselves pushing that narrative relentlessly.


    Nor do ICM come up with any reasonable explanation as to why they start hunting for shy NO voters when the polls are narrowing and back when they showed with a good poll for Yes. (as if that in itself was somehow too preposterous to be believed) If this phantom menace of shy No voters is real and so 'obvious' or 'intuitive' then they should have been doing it long before now to stop it looking merely like the latest excuse for the No campaign.


    The bizarre placing of the dodgy "comfort" question first is now tainting the very thing a pollster might be expected to do competently. The actual indyref question itself. Well played ICM, well played.

    The only person who comes out of this debacle with less credit than ICM is of course John Curtice. It's hardly a surprise he wants to move on very quickly to talking about the May polling now that all this has been revealed. His silence on the matter speaks volumes and it's not as if he was seen as particularly impartial before considering how hard he used to push his own narrative of there being no change and no narrowing in the indy polling when there quite obviously had been.

    I fear Curtice next foray into indy polling will still be dogged by some quite pointed questions about this ICM poll and what hye didn't say about it, sadly for him. No wonder he and ICM have got a place in the hearts of many a No voter, because they tell em what they want and need to hear.

    As for those No supporters trying to take 'comfort' from this obviously dodgy and ill-thought out attempt to find shy No voters, good luck with that given how badly it was conceived and executed.

    FWIW I think (based on my own anecdotal research) it's hugely overblown and despite there being a few shy Yes and No voters it really doesn't amount to anything like the gamechanger the No campaign were self-evidently hoping it would be.

    ReplyDelete
  12. "The fact that No rose after the interviewee was asked about being uncomfortable is because No-voting interviewees examined their mood and worked harder to overcome their higher discomfort after being made aware of it"

    That sounds a hell of a lot more 'creative' than my own interpretation!

    If I had answered the 'comfort' question by saying that I was uncomfortable, my first reaction upon being asked the referendum question only a few seconds later would have been extreme annoyance that I had been asked for my wishes and they weren't being taken into account. That would have made me feel that the pollster was less 'sympathetic to my plight', not more. The only way round that problem would have been for ICM to produce a separate screen for uncomfortable respondents reassuring them that the survey is completely anonymous and that they need have no fears about answering honestly. For all we know they may have done that, but there's certainly no evidence that they did.

    ReplyDelete
  13. BTW James if you are looking for a laugh have a quick look at recent PB threads. The hilarious spectacle of racist tories shrieking at the top of their lungs about racist kippers is pure comedy gold.

    The incompetent PB tory moderators aren't even pretending to be even-handed any more and are pretending this is all light-hearted 'trolling' of the kippers. Not that the kippers and Farage don't look pretty comical themselves, yet it somehow doesn't occur to the out of touch PB tories and the right-wing press that they might not actually be best placed to start lecturing others over immigration and racism.

    "Are you thinking what we're thinking?"

    LOL

    ReplyDelete
  14. I take it your 'technical fault' banning still hasn't been lifted, Mick?

    ReplyDelete
  15. @James

    +Checks+

    Nope.

    LOL

    I can't say I've expected it to be James. The sheer unbridled lunacy and cowardice we saw in the 'explanation' that TSE was cornered into giving to theuniondivvie didn't really make it seem that likely.

    Another point of huge amusement is that Stuart has replaced Tim as the politicalbetting's chief (and most times only) betting poster. The PB tories and kippers seemingly locked in a ferociously amusing battle of the idiots while left of centre opinion is noticable by it's absence. As usual.

    I hope you don't mind me intruding into an interesting polling thread with the latest PB tory comedy.

    BTW even UKPollingReport were very far from being convinced by the latest Mori debacle or the Curtice narrative as you can see here.

    http://ukpollingreport.co.uk/blog/archives/8815/comment-page-1#comments

    ReplyDelete
  16. Oops! "by the latest ICM debacle" that should have read.

    ReplyDelete
  17. What I want to know is are people connected to these polling companies, lumping big money on a Yes vote? as odds on Yes have been getting slashed, it would be quite a nice earner if you knew that bookies used polling results to set odds and so were able to be part of a group who both manipulated polling results/odds and put money on these odds. people can make a lot of money from gambling if they can manipulate the odds.

    ReplyDelete
  18. FWIW I think (based on my own anecdotal research) it's hugely overblown and despite there being a few shy Yes and No voters it really doesn't amount to anything..

    Of course, my own thoughts don't preclude a bit of a shy vote on either side. However, the results suggest to me it's a larger for Yes. Not massive, but there, even in online polls. It's not just this poll of course, but e.g. the way Yes decreases markedly in TNS face to face but no stays the same amongst 2011 voters as I noted.

    As for a huge shy No...
    Frankly, I find the idea of hordes of people who don't like Braveheart, dislike Salmon, don't want separation / to break up the wonder family of nations that is Britain, don't want chucked out of the EU and forced to adopt the Euro, are clearly not anti-English and not economically illiterate being afraid to admit this to British/UK polling companies laughable.

    ReplyDelete
  19. Just my thoughts on this:

    It makes sense to me that ICM's figures are now most similar to TNS's considering they now use the same weighting procedure.

    However what I would say is something else might be going on here, because ICM definitely have been the most volatile pollster, even without the methodological changes.

    I have said before my doubts about how representative online volunteer panels can be, after all I believe I have taken part in every single Panelbase/Sunday Times poll. Chances are a lot of online samples are political anoraks being interviewed over and over (like me). While that probably works for an election, I don't believe it does for a referendum.

    I genuinely believe that TNS BMRB will end up being the most accurate on the day. Between the fact they are interviewing truly random and ordinary members of the public and their weighting procedure represents the whole electorate (Although I know you don't agree with that James).

    ReplyDelete
  20. James,
    Is there any evidence that opinion polls have an influence over how people actually vote on the day? In general elections,for instance?

    ReplyDelete
  21. If we exclude SNP voters who are clearly the most pro-yes and comfortable admitting their intention for obvious reasons (e.g. they are not going against their own party and that party won a landslide leading to the referendum), we see an interesting correlation.

    Amongst 2011 voters of other (namely pro-UK) parties and those who did not vote, the more pro-Yes the group is, the more uncomfortable people from it are with expressing an indy ref intention.

    Of course conversely, the more support for No in the group, the more comfortable people in that group are with giving an intention.

    Implies shy Yes, not shy No.

    Most Yes / Most uncomfortable about giving an intention in descending order:
    - Did not vote in 2011
    - Other party voters (will include iGreens)
    - Labour voters
    - Lib voters

    Tories bottom of the list and are totally fine with expressing their strong No intention.

    ReplyDelete
  22. Sunshine on CrieffMay 21, 2014 at 1:45 PM

    One thing that puzzles me is that ICM is an online pollster. Surely the anonymity this brings does away with any 'shyness'?

    I could understand it more if people were shy in face to face or telephone polls, but online - why?

    Also, given that TNS BMRB (face to face) and Ipsos MORI (phone) are the most Yes unfriendly pollsters, doesn't that point to Yes people being more shy?

    ReplyDelete