Concerns raised over Ipsos-Mori's methodology

An anonymous poster left this rather troubling comment on last night's post about polling accuracy -

"James this is an excellent article and has prompted me to share with you my own personal experience.

I had never been polled before but in September I was contacted by Ipsos Mori. On paper I should be a No voter, retired professional, own my house, good company pension etc.

After being asked if I would take part in the poll, I was then asked if they could contact me on a regular basis as they wanted to follow trends.

After two or three very leading questions, one of which started "since Scotland would not be admitted to the EU", I was then asked "if the referendum was tomorrow how would you vote" answer Yes. "How likely are you to vote" answer definitely.

After a few other questions the interviewer thanked me for my time then finished the call.

I have never heard back from Ipsos Mori, I wonder why?"

Now, to be fair, it's possible that this was an unpublished internal poll for the No campaign or for one of the anti-independence parties. If so, there's no problem at all - if Better Together are daft enough to use leading questions to convince themselves that they're doing better than they really are, then by all means let them get on with it. But there was in fact a published Ipsos-Mori referendum poll that was conducted between the 9th and 15th of September last year. There was no suggestion at the time that any questions (let alone leading questions) were asked before the main referendum question, and if there had been it would have completely transformed our perception of the results - look at the way Professor John Curtice castigated the Panelbase poll just one month earlier that had asked the referendum question third.

Because this was a telephone poll, one theoretical possibility is that one or more rogue interviewers were asking the questions in the wrong sequence, and thus potentially distorting the results. It's not entirely paranoid of us to raise these doubts, because as things stand Ipsos-Mori are the extreme outliers in this campaign, and that must be happening for a reason (or for a variety of reasons). The other slightly mysterious thing about them is that they did actually produce one poll that was very good for the Yes side, way back in early 2012. That was followed by a huge slump for Yes over the course of the rest of the year, from which there hasn't really been a proper recovery in any subsequent Ipsos-Mori poll. No other pollster has replicated that trend. The only one that has even come close is TNS-BMRB, but unlike Ipsos-Mori they've shown Yes making a very telling recovery from the low point of the initial slump.

It's very difficult to explain this divergence from the general pattern, given that as far as we know Ipsos-Mori haven't altered their methodology since that early 2012 poll. So if anyone else has been interviewed by them, feel free to let us know your own experience. Were you contacted on your landline phone, or on your mobile? Was the referendum question asked first? If not, could the earlier questions be construed as in any way leading? Was there a preamble to the referendum question, and if so, what was it?

UPDATE : I've been contacted by someone else who was interviewed by Ipsos-Mori more recently. He asked to be anonymised, so I've excluded his postcode and a few other details that might conceivably be personally identifying. In some ways it's quite a similar account to the one above, but crucially there's no indication this time of any leading questions being asked prior to the referendum question. There's still no sign of anyone having been contacted by Ipsos-Mori on their mobile phone. That's important, because if they aren't calling mobiles it's hard to see how they can be confident that their sampling isn't significantly skewed.

"Firstly, thank you for your excellent blog, which has helped me to come to a dim appreciation of some of the complexities of opinion polls. I'm stimulated to contact you as a result of reading your post this morning, with details of a reader who contacted you with details of his experience with IPSOS-MORI.

I have to say that I can echo that experience. I was contacted by them by telephone (my landline) earlier this year - I would guess around 6 to 8 weeks ago, but add a significant plus or minus on either side of that. I was asked a similar set of question (no leading comments re the EU, but was asked my voting intention, which is Yes, and my likelihood of voting, which is certain). The final question which I was asked related to age - when I said that I was **, I was told that their quota...was filled, and that they could go no further with questions. I was asked if I would be happy to be contacted for future polls, and answered yes, but no further contact has been made to date.

Interestingly, I fall into the same demographic as your other correspondent...I live in a fairly affluent area...and guess I should be a right of centre No voter (neither of which tags describes me!). I don't know if these two anecdotal reports have any significance, but it seemed worth giving you this information."

If you scroll down to the comments section, you can also find an interesting comment from an ex-Mori interviewer.

* * *

I did actually manage to make it through the whole of Clegg v Farage this time, but as someone with pro-European views it was a profoundly depressing experience to have as my supposed "champion" a politician as unappealing and insufferably condescending as Nick Clegg. I'll be honest - he was so dreadful that it got to the point where I was more or less cheering on Farage. In retrospect, it's hard to understand why the public didn't see straight through Clegg in the 2010 leaders' debates, because his style hasn't changed one iota - it must have been a very weird kind of novelty value.

Instead of treating us like adult human beings and saying "that's because of the Lisbon Treaty", he'll say "that's because of something called the Lisbon Treaty, which is...", before proceeding with a patient explanation in words of no more than one syllable for his very favourite little girl Hannah, who asked such a clever question. In fact, I almost expected him to say : "That's a really important question, Hannah, and I'm going to answer it for you with what we call 'words'. Those are really cool things that we use to make up a sentence..."

You won't be surprised to hear that the bit that made the most steam come out of my ears was when Clegg abused the platform he'd been given by embarking on an anti-independence rant. Dimbleby did eventually close down the topic, but he waited far, far too long - it felt like a good thirty seconds went by. And even then, the way he did it was totally unsatisfactory - he seemed to imply he was only shutting Clegg up because independence was a rather tiresome subject. Instead, he should have said this -

"Nick, stop. This isn't on. You know perfectly well that there is no representative here from the pro-independence campaign to put the alternative point of view. The reason they aren't here is that they weren't invited. The reason they weren't invited is that independence isn't on the agenda for discussion tonight. So it's inappropriate for you to raise it, and I'd ask you not to do it again."

The point being of course that Clegg knew full well that he was chancing his arm, and if other anti-independence politicians are to be deterred from doing the same on network TV they need to know they'll be clearly 'punished' by having their transgression flagged up for viewers. A moderator half-heartedly changing the subject after the damage has already been done just isn't good enough.

Interestingly, the BBC's own guidelines call on presenters to effectively slap down politicians who try to slip in a sly anti-independence dig in interviews related to other topics. Those guidelines won't officially come into force until late May, but Dimbleby's performance tonight doesn't fill me with confidence that they'll ever be properly implemented.