Concerns mount over ICM's sudden change in methodology

The datasets for ICM's referendum poll have been published, and as a result it's become clear that the methodological changes have gone way beyond the one flagged up by John Curtice and trailed by Martin Boon last month, namely the introduction of a likelihood to vote filter.  In the April poll from ICM that produced a virtual dead heat (a No lead of just 3%), the referendum question was asked first.  But in this new poll showing a bigger 12% No lead, the referendum question was asked third - which ought to set alarm bells ringing immediately, because we know that responses to later questions can be substantially influenced by the wording of earlier ones.  Indeed, John Curtice has spent a fair bit of the last few months rubbishing a poll conducted last summer by the BPC-affiliated firm Panelbase, simply on the basis that it asked the referendum question third.  That poll of course showed Yes in a 1% lead.  A few cynics might wonder why he pounced on methodological bad practice in a poll that showed movement towards Yes (and in retrospect he was probably right to do so, because it was completely out of line with other polls), but hasn't said a word about the same problem in a poll that appears to show a favourable trend for No.

It is of course necessary to look at the wording of the two questions that precede the referendum question before jumping to the conclusion that they might be distorting the result.  The second question is entirely innocuous - it simply asks how certain people are to vote, and as that's a dry, technical question there's no problem with asking it before the main question (other pollsters do the same).  But the first question is a different matter entirely.  It asks -

"Thinking about the referendum on independence for Scotland, do you feel comfortable or uncomfortable being asked which way you might vote?"

Frankly it's hard to think of a dodgier question to ask right at the start, because when respondents come to the referendum question a few seconds later they'll be thinking to themselves, "ah, this is what some people are uncomfortable about being asked", which is bound to influence their reaction in at least some cases. Indeed, a fair chunk of people will have already indicated that they ARE uncomfortable about being asked, which arguably gives them implicit permission to do one of two things - to answer 'Don't Know' even though they do know how they will vote, or to give a completely dishonest answer.

If there's been an explanation offered for the introduction of the 'comfortable' question, and for its positioning within the question sequence, I haven't been able to track it down so far. My best guess is that this constitutes ICM's 'investigation' into the possibility of 'Shy No Syndrome', which Martin Boon revealed last month was on its way. It was already a matter of huge concern that ICM distrusted their results simply because they were so favourable for Yes, and seemed to be actively looking for excuses to adjust the Yes vote downwards, rather than open-mindedly testing for any bias that might be occurring in either direction. Even so, I naively assumed that they would be able to carry out their 'investigation' in a manner that did not compromise the integrity of this month's poll, but sadly that does not seem to have been the case.

So what did this experiment discover? If the point of asking for people's comfort levels is the assumption that some of them are answering dishonestly due to social pressure, what ICM would have been looking for to prove their theory about 'Shy No Syndrome' is evidence that a greater number of people who say they are voting Yes are uncomfortable about being asked the question. In fact, it turns out that the opposite is true. No fewer than 70% of people who say they are voting Yes are "very comfortable" with being asked the question, compared to just 52% of people who say they are voting No. Just 5% of people who say they are voting Yes admit to being uncomfortable, compared to 9% of people who say they are voting No. Whether this disparity came about because ICM effectively gave people 'permission' to feel uncomfortable, and as a result a chunk of respondents who would otherwise have said Yes switched to No (thus distorting the headline referendum figures), is anyone's guess. But given that this very peculiar methodological change coincides with an increase in the No lead that two other pollsters have failed to replicate, it's an obvious suspicion to raise.

Scottish Skier suggested that I should remove the ICM poll from the Poll of Polls due to this bad practice. I can't really do that, because the whole point of the exercise is to throw all the wildly varying methodologies from BPC pollsters into a pot and see what the blend produces. And this certainly isn't the first time that a BPC pollster has used dodgy methodology - remember YouGov's embarrassingly biased preamble about Scotland "leaving the United Kingdom"? But it's undoubtedly a big concern, and it may well be distorting the trend in the Poll of Polls. The best way of judging the real trend for now is to look at recent polls conducted without a change in methodology - and TNS-BMRB, Panelbase and Survation are all showing a relatively stable position, with Yes consolidating the big gains made over the winter.

Incidentally, there are a couple of pieces of good news from the ICM datasets - the Yes vote on the unrounded figures after Don't Knows are excluded is 42.4%, which is obviously a touch higher than the rounded figures for publication suggested. And as seems to be the case every month, the Yes vote rises a touch (this time to 43.2%) after Don't Knows are asked how they are leaning.