Thursday, May 29, 2014

Problems with the BBC referendum poll tracker

The BBC website has introduced a new poll tracker for the referendum.  It uses a format that in many ways I quite like - it doesn't compare apples with oranges by lumping together polls from different firms in the same graph, and instead simply provides a separate tracker for each of the six pollsters.  And it very sensibly excludes Progressive Scottish Opinion, although interestingly it does so on the grounds of the question PSO used in their most recent poll, as opposed to the fact that they do not abide by British Polling Council rules.  But there are still two big flaws that leap out at me -

1) When you first visit the page, all you can see is the ICM tracker, and yet there is no explanation offered that you're only looking at polls from one company - casual visitors will be forgiven for assuming that they're looking at a graph showing all the polls from the campaign so far.  You really have to study the page before there's much chance of spotting the drop-down menu which allows you to navigate between the trackers for the six different pollsters.  This is a particular problem because the ICM tracker is the most misleading of the lot, due to the endless methodological tinkering that the firm have indulged in on a virtually monthly basis.  Leaving aside the open question of how big an impact their bizarre introductory question had in the most recent poll, one thing there can't be any doubt over is that their recent decision to start dramatically upweighting responses from people who didn't vote in the 2011 election has increased the reported number of undecided voters, meaning that the headline Yes vote (and indeed the headline No vote) can't be fairly compared with earlier polls.

2) The Panelbase tracker shows just two polls this year, both commissioned by the Sunday Times.  In reality, Panelbase have conducted six referendum polls this year - the other four were commissioned by the SNP, Newsnet Scotland, Wings Over Scotland and Yes Scotland respectively.  I can only assume the explanation for this omission is supposed to be covered by this sentence -

"It does not include polls conducted on the behalf of any political parties."

OK, that explains the exclusion of the SNP-commissioned poll, and at a stretch it might explain the omission of the poll for Yes Scotland, which has strong links to the SNP.  But Newsnet Scotland and Wings Over Scotland are alternative media websites that are entirely independent of any political party.  True, they both have a desired outcome in the independence referendum, but then so do many mainstream newspapers that commission opinion polls.  Such as the explicitly anti-independence and Labour-supporting Daily Record, for example, who commission the monthly Survation poll series.  Are those Survation polls excluded from the BBC tracker on the basis of the same logic?  No they are not.

Would the BBC exclude polls commissioned by the even more virulently anti-independence Daily Mail?  Almost certainly not, you'd have to conclude.  So just how partisan does an anti-independence publication have to get before it's deemed by the BBC to be "out of the game" in the same way as Newsnet Scotland or Wings?  Could it be that there's one law for Yes-supporting media, and another law for No-supporting media?


  1. I just put a link to this site on Wings after a reference to the BBC Poll a comment,

    Then I wentt on to read the next wing article which had at the start a statement about people only reading parts of articles.

    Here’s an important statistic: almost 40% of people who look at an online article don’t get beyond the headline and strapline. More and more readers fall away the further down the article you get – by the time you’re just a few hundred words in, you’ve probably lost roughly 70% of people who started reading.

    See point 1 of James's comments

  2. Yes it is one law for the Yes-supporting media and another for the No-supporting media.

    I really thought you were paying attention, James!