Thursday, March 12, 2015

Now that the pollsters are an official part of the democratic process, they can't act as if their methodology is a private matter for them alone

My unexpected Twitter encounter with YouGov's Laurence Janta-Lipinski a few hours ago refreshed my memory of the exchange I had with him during the height of the referendum campaign last summer.  Basically what happened was that Laurence was in a grumpy mood because such a large number of intensely stupid people were refusing to recognise that YouGov were so obviously superior to other pollsters - he posted two or three generalised tweets along those lines, and it was pretty clear he was casting around for someone to make an example of.  As it happened, I had just written a post on this blog referring to YouGov's lack of transparency, and so I was chosen as the lucky victim.

Laurence challenged me to explain how YouGov were less transparent than other BPC pollsters, which was plainly supposed to be an unanswerable question.  But unfortunately I did have an answer, namely that they were keeping their "Kellner Correction" figures secret - which meant that we were unable to confirm our strong suspicion that the Correction was artificially boosting the reported No lead in the firm's referendum polls by a significant amount.  Instead of acknowledging that I had specifically answered his question, Laurence then started harrumphing about how, of the many unjust complaints that poor, put-upon YouGov have to listen to, criticisms about secrecy were a new one on him.  So I challenged him - if he regarded his firm as being so transparent, would he now commit to releasing the Kellner Correction figures in full?  And if not, why not?  It turned out that his passion for answering questions was somewhat less strong than his zeal for asking them - he suddenly pompously declared that my question was much less important to him than looking after his sick one-year-old son.  (Incredibly, he used almost exactly the same words again to close the exchange a few hours ago, albeit this time it was his son's dinner that was more important.)

Now, let me just step back at this point and make the obvious observation.  I have no problem acknowledging that parental duties must trump everything else, but you know what?  The beauty of social media is that you can respond whenever you like.  You can disappear for seventeen hours, or three days, and then still answer the question, and you certainly don't have to account for what you were up to in the interim.  It's very, very hard to escape the impression that the only reason Laurence randomly mentions his son when the going gets tough is the hope that any attempt to point out that he keeps evading perfectly reasonable questions can somehow be painted as a violation of his family life!  In any case, he presumably already knew his son was sick at the point at which he decided to initiate the exchange, and it also has to be said that it would have taken no longer for him to actually answer me than it did for him to haughtily philosophise about the relative importance of my question and his loved ones.

Anyway, I didn't let him off the hook, and eventually he angrily told me that no, we wouldn't be getting the Kellner Correction data, and the reason (or non-reason) was that YouGov didn't have to do things at the behest of mere bloggers.

We had every cause to be hopping mad at the time about that display of arrogance, because we knew that YouGov's chief Peter Kellner carried a good deal of baggage on the subject of independence - he had been saying for years that it was quite literally impossible for Yes to win, and therefore the obvious suspicion was that he was (perhaps subconsciously) moulding his methodology in a way that would "prove" himself right.  It wasn't as if these polls were being released into a sealed antechamber - they were fundamentally affecting the reporting of the referendum campaign, and essentially sucking the life out of it.  Until a very late stage, there were many anecdotal reports of people saying that they were just going to vote No by default without thinking about the issues in any great depth, because they already knew what the result was going to be.  That was - at least in part - YouGov's handiwork.  If Kellner and co reserve the right to play God with the future of our country in that way, I don't think it's good enough for them to turn around and say "we don't have to explain ourselves to the little people".

But that point has even more force in the context of the general election campaign, because both Ofcom and the BBC have partly based their decisions over the apportionment of TV and radio coverage for each party on the basis of opinion poll data.  That makes the likes of YouGov practically an official part of the electoral process - and whether they like it or not, that means the question of whether they're getting their methodology right is not just a private matter for them.  Any lack of accuracy on their part has the potential to directly affect who governs us for the next five years, and therefore we have every right to demand maximum transparency, and to ask searching questions.

I'm not sure Laurence Janta-Lipinski has got that memo yet, though. His latest excuse for never having properly answered my question about the Kellner Correction was that I had "edited out a wink smiley" when I quoted him, which was "very dishonest".  He attempted to grill me in Paxman-esque fashion this evening about what my despicable motive had been for excising the smiley.  Seriously.  This is a thing that happened.  As before, though, I only got a "does not compute" message when I dared to ask him a question of my own.

14 comments:

  1. Stick it to him, James. Not overly enamoured of that breed.

    Had a point of dispute with the Ipsos Mori suit early on indyref day. Made a misleading (false) statement (that could effect voting behavior) that appeared in every paper, around late morning/lunchtime.

    He first claimed there was no error, then after much pressure, that there may have been an error but it was a mere bagatelle with no substantive impact, and therefore he would not correct it. It was evening and more banging on the drum before he decided that indeed it may have an impact and at last corrected it. But by then of course, any damage would have been already done.

    Given the margin on the day, it would not have changed the result at all - not even close. However, at the time of its original dissemination we didn't know that. In a real close run contest it could have made the difference between win and lose.

    I can only think of two explanations - negligence rising to the level of professional malfeasance or a conscious choice to inject the "unfortunate misstatement" into the news cycle.

    ReplyDelete
  2. Speaking of YouGov, their sub-sample result today is more like the long-term average, after a few days of going below then above.

    SNP 43, Lab 27, Con 18, Others <4.

    http://d25d2506sfb94s.cloudfront.net/cumulus_uploads/document/sjv8qo2081/YG-Archive-Pol-Sun-results-110315.pdf

    ReplyDelete
  3. I notice in that Redbox Yougov left-right survey that Scots seem to see themselves as quite measurably more left than people in England. Given that's a big topic of 'debate' for the coming election, surely the BBC should follow up with a full Scottish poll and devote a week to analysis of the results?

    ReplyDelete
  4. Here's a thought that might be worth a bit of discussion. Snp win a majority of seats (hopefully 50 plus) they gain agreement for FFA or as close to it as possible, with borrowing powers. Will they agree an EVEL trade off for it? If so, how would Labour react? This could potentially expose Labour even more, as being power hungry and having no real interest in Scotland? Thoughts?

    ReplyDelete
  5. Looks like the Lib Dems have been doing some of their comfort polling in Scotland as well. They've produced a poll, with fieldwork done by Survation, of West Aberdeenshire showing them narrowly ahead of both the Tories and the SNP. Most likely it will have the same characteristics as the poll they produced in an English seat earlier this week. Question ladder with the MP's popularity rating asked before the VI question, full weighting back to the 2010 result and sample size of around 400.

    https://www.pressandjournal.co.uk/fp/news/aberdeen/517422/polling-suggests-lib-dems-can-retain-north-east-seat/

    ReplyDelete
    Replies
    1. Fieldwork done at the same time as Ashcroft, so it isn't a question of opinion having changed since his poll.

      Delete
    2. "Question ladder with the MP's popularity rating asked before the VI question, full weighting back to the 2010"

      Almost meaningless then since you obviously don't get a question ladder in the polling booth. I'm surprised they have money to burn on duff polls in such a frivolous way. Not that they will have all that much in the way of activist numbers for the ground campaign either considering how massively unpopular calamity Clegg has made the yellow tories.

      Delete
    3. Mmmm, very sceptical of that. Rather small sample size and I'm always naturally suspicious of people releasing private polls as there's always the risk of cherry picking the most favourable results. Maybe at best it says the Lib Dems are possibly in with a shout, but nothing more.

      Delete
    4. It seems to be from the same set of polls that resulted in this.

      Survation were not responsible for drafting the questionnaires used, sampling design discussions or analysis of the results. These polls should therefore not properly be described as “Survation polls”. As a consequence, Survation is not responsible for the publication of these polls under BPC rules. Any member of the public with queries regarding the detail or further information about the mentioned polling work should be directed to the Liberal Democrats.

      http://survation.com/in-reference-to-recent-liberal-democrat-polling-shared-privately-with-the-media/

      So complete junk polling then. Zero credibility.

      I'd add that Clegg's ostrich faction would be a laughing stock to feature a set of disowned and discredited polling in any election literature but they're a laughing stock anyway so that was probably the plan all along.

      Delete
  6. The industry body who "regulate" them are made up of *and funded by*, you guessed it, the polling companies. I can't see that private companies who can somewhat skew the outcome of elections should be allowed to do so without a full transcript of their methods every single time they release figures. They still bandy the 3% error value around like it actually means anything, that's sampling error not the total error in their system which is totally unknowable, though we can guess. All of this leaves a lot of wriggle room for bad behaviour I'd say! I also bet a significant number of polling company employees have no idea about basic statistics and error, which makes interaction with them online in any sensible academic sense quite difficult.

    All of this is rather like getting The Portman Group to do the NHS stats on morbitity/mortality rates from alcohol consumption!

    ReplyDelete
  7. Doesn't take a polling geek to trash the shit out of the BBC's attempt at stirring up racial tensions + say 'Look: Scotland and England both don't like foreigners - vote Labour / No' through Yougov. Just as bad push poll questioning as that Survation one last year.

    http://www.scottishrefugeecouncil.org.uk/news_and_events/latest_blogs/2597_welcome_to_scotland

    Welcome to Scotland?

    Pauline Diamond Salim, Media Officer at Scottish Refugee Council

    Ask a blunt question, get a blunt answer.

    What a missed opportunity from BBC Scotland to tell us something we don’t already know.

    This week’s poll on Scottish attitudes towards immigration seemed designed to do one thing: generate a set of headlines. It certainly generated little insight into what people in Scotland think about immigration to Scotland...


    All the flaws pointed out succinctly.

    ReplyDelete
  8. Ipsos Mori sub-sample (very small size) produces a curious result:

    SNP 42, Con 29, Lab 14, LD 12.

    https://www.ipsos-mori.com/Assets/Docs/Polls/mar2015web.pdf

    ReplyDelete
    Replies
    1. Vote Labour and let the Tories in by the back door :-)

      Delete
  9. YouGov, eh.

    We know at least 2 things about them.

    1) Tory-supporting Murdoch media is a very lucrative client for them.

    2) They often show very favourable figures for the Tories at timely moments - particularly after big set-pieces - which the Murdoch media use to big up their beloved blue boys. Even if other pollsters show no similar shifts.

    I'm sure these two things are entirely unrelated, of course.

    But I'll be watching the polling on Gideon's upcoming make-or-break budget with interest.

    ReplyDelete