Monday, October 29, 2018

Are online polls superior to telephone polls?

You may have seen that Professor John Robertson has a blogpost about Saturday's Survation poll, which he regards as less reliable than a recent YouGov Scottish subsample, because he thinks "landline telephone" sampling is inferior to online sampling.  The post is actually based on a false premise, because Survation have confirmed today that their poll was conducted online - and it goes without saying that a full-scale online poll should be taken more seriously than an online subsample.  The confusion probably came about because of an ambiguously-worded tweet from Survation on Saturday evening.

Even if the Survation poll had been a phone poll, though, there would still have been a number of problems with John's argument.  First of all, although YouGov subsamples can probably be regarded as more credible than subsamples from other firms (because they appear to be correctly structured and weighted), they obviously have a bigger margin of error than full-scale polls because of the smaller sample size.  So to get a meaningful picture you have to look at the pattern over a number of YouGov subsamples, and it's pretty obvious that the SNP's 47% showing in the subsample John is talking about is an outlier.  High 30s is much more typical - in other words pretty similar to what the Survation poll found.

Secondly, it's highly unlikely that Survation would conduct a landline-only phone poll, so the concern John raises about certain demographic groups being less contactable by landline doesn't really apply.  It may be that response rates to phone polls are unacceptably low because people these days are unlikely to answer an unexpected phone call, regardless of whether they're on a mobile or landline.  But that's a somewhat different point.

Thirdly, there's the standard Mandy Rice-Davies objection to the quote John provides from YouGov about the supposed greater accuracy of online polling.  YouGov are, and always have been, an online-only pollster, so "they would say that, wouldn't they?"

Fourthly, John points to the fact that online polls were much more Yes-friendly during the indyref.  But in fact there was a dramatic convergence between the online and phone polls as the campaign drew to a close, and by polling day they were more or less showing the same thing - a very, very slender No lead.  So it's impossible to know for sure who was getting it right earlier on.  Anecdotally, a lot of campaigners did detect a large swing to Yes in the closing weeks, which would lend more support to the theory that the telephone polls were more accurate.  (YouGov were the only online firm to report a big swing, and they only did so because of their notoriously convoluted "Kellner Correction".)

Lastly, John mentions a ScotPulse online poll showing a handsome Yes vote.  Unfortunately ScotPulse polls can't be taken seriously because they're not properly weighted.  The (allegedly) best data collection method doesn't really help much if the other basics aren't being done correctly.

*  *  *

25 comments:

  1. Last time I saw a >43% SNP in a Yougov subsample was back ahead of the 2017 election. However, yougov samples have shown a long term steady but modest increase in SNP share since a bottom out post 2017 election. So, while 47% looks like mainly noise, it comes on the back of movement in that direction and isn't ridiculous historically of course. However, it would be silly to get carried away about as you say.

    Survation at the lower end, Yougov subsample at the higher end of variance. Polling generally very steady for a year now, except for views on the UK getting a good brexit deal. They've crashed to ridiculous lows both here and in England. That's important as it could be a warning for large swings as we near England's big day.

    ReplyDelete
  2. You have to congratulate the government for getting us out of the corrupt dictatorial EU. The sight of SCOTTISH EU grovellers is shameful to Scotland.

    ReplyDelete
    Replies
    1. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia.

      Delete
  3. Am I right in thinking that there was a similar convergence in the results of online and phone polls in the EURef: with Remain's lead in phone polls becoming markedly less pronounced as voting day drew near?

    From the blogpost:

    "So, commonly, telephone surveys generate conservative, negative or status quo returns. Respondents are more likely to say no to a question about a big change of some kind."

    One possible explanation could be the inherent problem with online surveys: that they are chosen from a self-selected pool of people willing to sign up to answer YouGov's questions every so often. These people by nature will probably be more engaged/knowledgeable about/interested in politics than the average voter, and so will likely have devoted more time to thinking about the kind of questions they will be asked than the average voter will. Hence the phone respondent who has engaged with the questions less will be more likely to default to supporting the 'status quo' answer.

    But as the day of a big election/referendum draws near, the less-engaged phone respondents start to inform themselves and think about the questions more, and so their answers start to converge more towards the already-engaged online respondents: creating the pattern we saw in the closing stages of the Indyref.

    I'd be inclined to suggest the phone polls are more likely to be accurate on the grounds that most of the population is less interested in day-to-day politics than regular visitors to a politics blog.

    ReplyDelete
  4. Frua Merkel (thanks British mugs for subsidising uz former Nazis) ve took ye tae ra cleaners.

    ReplyDelete
    Replies
    1. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia.

      Delete
    2. They know. Act normal.

      Delete
    3. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia and its desperate need for human contact.
      So sad.
      So very funny.

      Delete
  5. I see the English Tory brexiters are already slashing taxes for the wealthy. Jeez they move fast; we're not even out of the EU yet.

    I wonder if Corybn and co will vote for their brexit bill? Could decide the final fate of Labour in Scotland and the union itself.

    ReplyDelete
    Replies
    1. You hope. Wonder if the Scots will ever know how much money they poured into the EU Mafia. OH they did give us some EU frozen butter during the eighties. WE WERE SOOOO GRATEFUL.

      Delete
    2. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia.

      Delete
  6. Bloody hell what a battering I got there. I surrender.

    Prof John

    ReplyDelete
    Replies
    1. My opinion (FWIW) is below John. ;o)

      Delete
  7. Will share with my 'people'. Lifelong learning!

    ReplyDelete
  8. Personally I think engagement, observance, cognizance and intuition are as reliable as any poll. I receive all "sensational" poll results with a wry smile.

    Polls are as useful as a one legged man in an arse kicking competition.

    ReplyDelete
    Replies
    1. Racist! I have great respect for the people of Poland.

      Delete
    2. They're not taking us seriously Bryan.

      When I was wee (1950s) my dad took me for haircuts to Alec the Pole. I thought it was because of the red and white striped pole he had outside his shop.

      Delete
  9. There's a telephone poll in our garden.

    ReplyDelete
  10. I commented on an earlier post about the NY TIMES_ Siena college doing a hundred polls of different races all over the USA , they are posting EVERY CALL online so you can see and Follow along as each call gets made, answered, not answered, etc. Some are still going on, they start one or two per day and take about three days. Some problems have been how many calls it takes to get an answer: sometimes in the hundreds to get one response. Often after 300 responses( out of 350 to 500 needed) there are HUGE subsets nearly vacant. Other than millennial there seems no reason why a subset in a particular district is not being reached. Another huge problem is how far the swings are within that total call #. Often a candidate is up 60% to 40 after 300. Down by 10 after 400. Up by 10 at 500. The PROBLEM is you could seriously , scientifically have stopped at any of these points and published the poll in good conscience ( in the full sub- set ones) . this NEVER used to happen.it raises the question of how many pollsters spend the extraordinary amount of money to keep polls in the field to the PREDETERMEND target? It is hard to get info out of most anyway afterwards. When you see poll response numbers vary, you have to wonder if this is effecting results.if they get 1,200 responses in one poll, but stop at 1.137 the next but go to 1,257 the next you are skewing the results? How many pollsters will spend 33% more to come back a fourth day to " finish" ? I bet a lot stop at a convenient point unless it's right before the actual vote. Yes 55 no 40 a month out? Similar result to expectations and other polls? Yes. Got 1,130 responses, wanted 1,300 and Scottish , Yorkshire. And 18-30 voter sub-samples short? Spend 33% more to fill it out AT POLLING COMPANIES EXPENSE? I doubt it. This may be why they are so vague at first. Three days later there are like 10 people in the world looking at the criteria.I would love to see a UK poll with full subsets and a preannounced total sample.

    ReplyDelete
    Replies
    1. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia and its nonsensical drivel.

      Delete
    2. Oh Cordelia, what a heavenly hat.

      Delete
  11. Scotland's independence drive is dying from chat. No independence movement in the world made decisions based on polls. They went for independence, with leads that worked for independence.
    Our leaders have been working to make Westminster decisions work in Scotland in the least harmful way.
    In the end that doesn't win anything.

    ReplyDelete
  12. So Labour back the latest Tory tax cuts for the better off. Why am I not surprised.

    We just need Corbyn to stand shoulder to shoulder with the Tories by voting for their brexit bill and voting Labour becomes utterly pointless.

    ReplyDelete
    Replies
    1. The Tories have lowered the tax threshold on four occasions since coming to power for lower earners. The Tartan Tories have tax raising powers but are scared to use them. The Tories are always ahead of the game and always backfooting their Tartan pals over tax.

      Delete
    2. GWC AKA The Hon. Cordelia Bracely-Dubois of the 77th (Manky Shirt, Self Funded) Auxiliaries and its unrequited love for Yaxley-Lennon.
      Poor, miserable, incoherent Cordelia and its nonsensical drivel.
      Always funny watching its various identities bickering with themselves.

      Delete