Friday, August 14, 2015

Challenging the revisionist history of referendum polling

There seems to be a mythology springing up about the polls in the run-up to the referendum last year.  This is from Andrew Rawnsley in the Guardian, setting out why he thinks the SNP may be cautious about rushing into indyref 2 -

"They did lose last time and just one poll in the entire campaign put Yes ahead and then only by a nose. There has not been a poll since showing a sustained majority for independence."

That's eerily close to what Lallands Peat Worrier said in a blogpost the other week (parts of it are almost word-for-word identical), so I don't think there can be much doubt where Rawnsley is getting it from. But it was factually inaccurate when LPW said it, and it remains factually inaccurate now.

There were in fact at least TWO polls that put Yes ahead during the campaign, and arguably there were as many as four, depending on how you define the words "campaign" and "ahead". The two definites were -

1) YouGov poll on Saturday, 6th September 2014, showing : Yes 51%, No 49%

2) ICM poll on Saturday, 13th September 2014, showing : Yes 54%, No 46%

The latter seems to have been edited out of people's memories, perhaps because ICM's Martin Boon cast doubt on its findings almost as soon as it was released. But nevertherless it was a real poll, it was properly weighted, it was published in a major Sunday newspaper, and John Curtice considered it credible enough to include in the What Scotland Thinks Poll of Polls.

The other two possibilities were -

3) TNS-BMRB poll on Monday, 8th September 2014, showing : Yes 50%, No 50%. (That could be said to have shown Yes in the joint lead.)

4) Panelbase poll on Monday, 2nd September 2013, showing : Yes 51%, No 49%.  (This one should be included depending on when you interpret "the campaign" as having started.  Professor Curtice invariably dismissed it as "a much-criticised poll from Panelbase" due to the use of an unusual question sequence.  But there was always a double-standard there, because ICM once did something very similar in a poll that was more favourable to No, without Curtice raising any objections.)

These are just the public polls - it's an open secret that what really spooked the No campaign was a private poll showing Yes 53%, No 47%.  And of course there were several public polls during the closing two weeks of the campaign that were 'statistical ties', ie. it was impossible to tell whether Yes or No were ahead due to the standard 3% margin of error.  In my opinion the most sensational public poll of the campaign was not the online YouGov poll on the 6th, but rather the two telephone polls from ICM and Ipsos-Mori that both pointed to a dead heat with figures of Yes 49%, No 51%.  The Ipsos-Mori poll was published by STV on the night before the referendum, and was bang up to date.

I presume the purpose of the "just one poll" myth is to promote the idea that the YouGov poll was a freakish outlier, and that there was never any serious evidence that Yes were in contention.  That argument simply doesn't stack up.

As I said in response to LPW, I'm also puzzled by this curious claim that "there has not been a poll since showing a sustained majority for independence". Clearly that implicitly acknowledges the reality that some polls since the referendum have indeed shown a majority for independence - but how can any individual poll show a "sustained" majority? Is the argument that there haven't been consecutive polls from the same firm showing a Yes lead? If so, even that isn't true.

50 comments:

  1. The shear panic from the No side in the closing stages wasn't down to one poll.
    The calling in of 'favours' from the Daily Record and Broon showed that the Empire was at serious risk. Project Fear blasted both BBC barrels to terrify voters because they knew how close it was.

    ReplyDelete
  2. 'how can any individual poll show a "sustained" majority?'

    Many possible definitions, but here's one: a majority which exceeds the margin-of-error, as part of a series of polls by different pollsters which also meet those criteria.

    However, I think there is too much nitpicking here. Indyref2, whenever it happens, will see another barrage of FUD from London and the MSM, which the indy movement will not be fully able to counter. For all practical purposes, a "sustained majority" needs to be one large enough not just to cover the margin of error (which is wider than the pollsters acknowledge), but big enough to survive some slippage in the face of an establishment onslaught

    ReplyDelete
    Replies
    1. "Many possible definitions"

      I disagree - I don't see how the word 'sustained' can possibly be applied to the result of any individual poll, and it certainly doesn't have anything to do with the margin of error (that's to do with size of lead, not how long it lasts).

      Delete
    2. I grant that it's poorly worded, but if you're not deliberately seeking out something to pick at, it can be taken to mean "polls since the referendum haven't shown a sustained Yes majority".

      Delete
    3. Neither have they shown a no majority.

      Delete
    4. Actually, something like the last ten polls have all shown a no majority - the latest one being in July, well after the conservative general election win.

      Delete
  3. Due to the nature of the proposition, the Yes campaign has to be consistently well out in front in order to win. A percentage of yes leaning voters will always bottle it and either vote no or simply not bother. So a razor thin lead in two or three opinion polls over the closing stages of a campaign will never be a sufficient predictor of a Yes victory.

    ReplyDelete
  4. Also, the 8 point ICM lead came from a poll of about 700 people - far fewer than the usual 1000+, thus increasing the margin of error.

    ReplyDelete
    Replies
    1. As I noted at the time, it increased the margin of error from 3% to less than 4% - hardly a huge difference. The sample size was also bigger than two of the three Angus Reid polls, none of which seemed to attract any criticism.

      Delete
    2. "As I noted at the time, it increased the margin of error from 3% to less than 4% - hardly a huge difference."

      We really need to stop throwing comments about the margin for error out there without mentioning confidence levels. There are two components of polling accuracy (margin of error and confidence level) and citing one without the other is meaningless.

      The theory will tell you that if you want a 4% margin of error with a 95% confidence level then you can get away with a sample of around 600. If you want a 99% confidence level at a 4% margin of error you need a sample of over 1,000. But this isn't an exact science and a 95% confidence level essentially means if you ran the same poll 20 times at least one of those would be even more inaccurate than your margin of error. Ultimately there's a very good reason why the standard polling sample has settled on somewhere closer to 1,000 rather than 700.

      So it's hardly surprising that a poll which had a lower sample size than is standard and produced a result that was out of synch with the other polling on the topic was regarded as an outlier. The mass of evidence we have from other polls and indeed the final result of the referendum shows that it was almost certainly correct to regard it as an outlier.

      It's also somewhat interesting that you knew fine well it had a lower sample size than the other polls yet chose not to mention that in your blog above. You also didn't mention that What Scotland Thinks only included it with a lower weighting in their poll of polls because of the lower sample size. Presumably you knew that as well yet chose not to mention it because it was inconvenient for your attempt at "mythbusting".

      As for the other two polls, the second one is 50-50. Given the pedantic nature of the argument you're making I think Mr Rawnsley would be well within his rights to point out that he said there was only one poll showing Yes *ahead*. He said nothing about "joint leads" - incidentally when your football team draws a game do you claim they "joint won"?

      The last poll is the well known Panelbase poll that was 1) conducted before the heat of the campaign; 2) methodologically flawed; and 3) completely out of keeping with every other poll at that time. So if you want to be strictly accurate there were three published polls showing Yes ahead, two of them with questionable methodologies that were rightly regarded as outliers at the time. At best Mr Rawnsley should have added the word "credible" before "poll" in his first sentence.

      Delete
    3. Rubbish. A poll with a 4% margin of error showing Yes on 54% is self-evidently just as significant as a poll with a 3% margin of error showing Yes on 53%. You're stretching a legitimate but small objection further than it will go. The issue is not that people might treat that ICM poll as an outlier - it's that they're pretending it doesn't exist.

      As for me not mentioning the fact that the poll was given a lower weighting in the Curtice Poll of Polls, you'll notice that I helpfully linked to the blogpost I wrote on the evening the poll was released, in which I discussed that very issue. I probably wouldn't have done that if I'd wanted to keep it a secret. Just a thought.

      The salient point is not the weighting Curtice gave to the poll in his Poll of Polls - it's the fact that he included it at all. He wouldn't have done that for something he regarded as "not a poll".

      Your quibble about "joint lead" (which in spite of your mockery is actually a very common term used in respect of political polls) is exactly why I listed it merely as a possible example.

      Your complaints about the Panelbase poll are much weaker. You don't approve of the methodology - that's fine, that your opinion, but it doesn't make it a non-poll. The only reason it's listed as a possible rather than definite example is because of the date, although from memory Curtice tended to regard that period as being part of the long campaign.

      I'm also not entirely sure what you mean about confidence levels never being mentioned. I believe I most recently mentioned confidence levels on this blog about two days ago.

      Delete
    4. My description of your argument is that about all it amounts to is a case for saying that Rawnsley's quote should have the word "credible" inserted into it - i.e. these polls exist but they have significant question marks over them. There was only one published poll with Yes ahead that didn't have a question mark over it.

      The point he's clearly trying to make is that there isn't much polling evidence to suggest Yes winning another referendum would be easy so the SNP are inclined to try and wait it out for more favourable conditions. The existence of two polls that nobody in their right mind would use to advocate a change in strategy makes essentially no difference to that point. Like all examples of pedantry it's technically correct, but utterly unsubstantial - and certainly doesn't justify the use of highly emotional language such as "revisionist history" given it's more than likely simply a minor oversight on the part of a journalist who doesn't obsess over the subject in the way you or indeed I might.

      Delete
    5. "i.e. these polls exist but they have significant question marks over them"

      That is simply not true. Pointing out that a poll has a fractionally - and it really is FRACTIONALLY - higher margin of error than most polls is not a "significant question mark". It's not even a question mark at all - it's just a very slight broadening of the health warning that applies to every single poll (or even a narrowing of that health warning, because a 700 sample is bigger than some polls).

      "There was only one published poll with Yes ahead that didn't have a question mark over it."

      Tripe. Say that a hundred times if you want, but it'll still be tripe.

      "The existence of two polls that nobody in their right mind would use to advocate a change in strategy makes essentially no difference to that point."

      I discussed no fewer than SIX other polls that cast doubt on the thrust of Rawnsley's argument. That's SIX, Chan. Count them.

      "Like all examples of pedantry it's technically correct"

      Are you for real? Pedantry? What? Pedantry would be if I point out a grammatical error in Rawnsley's sentence - not when I point out that THE ENTIRE CLAIM HE'S MAKING IS FACTUALLY WRONG.

      "given it's more than likely simply a minor oversight on the part of a journalist who doesn't obsess over the subject in the way you or indeed I might"

      If a journalist copying a false claim almost word for word from a blogger (without even bothering to take ten seconds to double-check on Wikipedia whether it is accurate or not) is your idea of a "minor oversight", I just despair. The Guardian must be glad to have as forgiving a customer as you.

      What is your own agenda, by the way? Forgive my cynicism, but it does seem rather probable that you have one of some sort.

      Delete
  5. A lot of this rhetoric, I think, ties into the fact that a lot of pro-indy commentators and politicians are worried about calls for a second referendum. There's a significant body of Nationalist opinion demanding a second referendum RIGHT NOW, which simply isn't practical.

    When folk speak about a sustained majority, I think, from a pro-indy perspective, we're speaking about a consistent 5-10% lead which would survive the next round of scare mongering from down south.

    You're spot on that multiple polls have shown us ahead in the past. What you're seeing from LPW is the heebie-jeebies associated with holding a poll too soon, losing again, and casting the movement into oblivion.

    Personally speaking, I don't want another referendum until we're absolutely certain of victory.

    ReplyDelete
    Replies
    1. That's an argument for never having another referendum, because absolute certainty will never be available.

      Delete
    2. I am wary of this argument too.

      I wouldn't say we have reason to be confident of winning if one was held now but the counter-point is that we might end up waiting till doomsday if people are too careful or want to see 60% consistently - that may never happen.

      I'm not going to pretend that it will definitely work this way again, but it needs to be borne in mind that the surge (for that's what picking up 15-20% from the starting base was) happened *after* the campaign kicked off in earnest (and, relatively speaking, towards the end of a long campaign, at that). Therefore, it *could* be that it we would only see a big shift in favour again when the issue became 'live' again. To avoid being misinterpreted, I am not saying here that those who are really into their politics, are still campaigning, and remain energised about it on a day-to-day basis don't consider it 'live' but there are a good many more people who will be interested in it but wouldn't consider it till it lands on their plates again. It's very hard to know what's right, but I think being too cautious could be as risky in its own way as being hasty.

      Delete
  6. I will make a prediction.
    No poll from a British company will (be allowed to) ever show Yes in the lead ever again, unless it is by a point or two to keep it deniable.

    ReplyDelete
    Replies
    1. Can we crowdfund our own poll?

      Delete
    2. Wings has done quite a few.

      Delete
    3. Yes, but unfortunately crowd-funded by people who don't realise or don't care that it is run by a transphobe. They wouldn't support it if he were a racist, but obviously transphobia is still socially acceptable, which is one reason why 41% of transgender people have attempted suicide at some point. (http://www.vocativ.com/culture/lgbt/transgender-suicide/)

      Natasha

      Natasha

      Delete
    4. Even if that were true, are you for real? A poll's not legitimate in some way because some of the people who funded it don't agree with some allegations you want to make? I fail to see how this would make the poll any more or less accurate. Pray tell, what polling organisations would be suitable? How would you ever ascertain that all of the people involved only held convictions you could support?

      Delete
    5. I'm not questioning the validity of any polls carried out by Wings, just pointing out that the person who runs it is a transphobe, and people should think carefully about whether they want to support him or not. And yes, I am for real; I have a transgender daughter to whom Stuart Campbell made the following comment on Twitter: "If someone looks like a man, then as far as I'm concerned he's a man and no sanctimonious wankhole is going to tell me otherwise." In other words, if my daughter isn't feminine enough to suit him, then she's not a girl.

      No wonder she doesn't like to leave the house, and is too anxious to meet strangers, with attitudes like that around. The Equality Act is now 5 years old; no one should think that comments such as that are acceptable in any way, shape or form. As far as I'm concerned, the fact that Stuart Campbell is transphobic far outweighs any contribution he has made to the independence movement; this should be a red line issue for anyone with a conscience. And I used to support Wings financially and in many other ways, so it's not as if I set out to look for reasons to criticise him. I was both shocked and disappointed to find that he is a bigot.

      Natasha

      Delete
    6. Getting back to Juteman's point. I don't know enough about psephology (or subterfuge) to know whether there's reason to doubt the ones we have, or whether its easy enough for a busy man like James to go ahead and commission one, but I'd certainly be interested in supporting one if it was on the agenda.

      Delete
    7. Humans are born either MALE or FEMALE. There's no such thing as trans-sexuals. Just people who have elective plastic surgery to make themselves conform to some demented body image.

      Would you rant about bigotry to somebody who despised plastic tits? No difference.

      Delete
    8. "but I'd certainly be interested in supporting one if it was on the agenda"

      I've thought about the possibility in an idle way, but at some point I'll probably have to run a third fundraiser just to keep the blog going, and I don't want to try people's patience by asking for money on top of that for a poll.

      Delete
    9. Anonymous, you should arrange to meet up with Stuart Campbell; you two would get along like a house on fire.

      Natasha

      Delete
    10. This is becoming stalking now. Ho hum.

      Delete
    11. The "Reverend" really is a strange figure. I mean, he lives in Bath, yet campaigns for Scottish independence. If he likes it in Bath, why doesn't he campaign for the union? Or if he is so passionate about Scottish independence, why doesn't he live in Scotland? I may be a unionist, no voter and tory. But at least I live here! When the SNP - or any other party in Scotland for that matter - upturns a fresh bucket of sh1t, I get hit with it along with everyone else in Scotland. The "Reverend" doesn't.

      Delete
  7. This obcession with polls is clearly a diversion from reality which is the lack of social policy from the Tartan Tories. Wake up Scots and bin the Nat sis.

    ReplyDelete
    Replies
    1. If you think polls are a waste of time, you should really stop hanging around here.

      Delete
    2. Just attempting to let you know there is life out and about on our planet not withstanding that a probe has arrived around Pluto although you probably arrived some time ago wee Jock fae ra North och aye ra noo innat.
      And what did ra Romans ever do for uz.

      Delete
  8. Polls need to come with a health warning. They are only accurate if A) people are telling the truth, and B) there isn't some seismic shift in public opinion between the poll and the actual polling day.

    I would say, however, that the Scottish independence referendum was as close to a foregone conclusion as you can possibly get. The finał result was a bit tighter than anticipated - but still a comfortable 'no'.

    And that is why there will not be a repeat until the 2020s, at the earliest.

    ReplyDelete
    Replies
    1. "I would say, however, that the Scottish independence referendum was as close to a foregone conclusion as you can possibly get."

      Now try saying it with a straight face.

      Delete
    2. It was always going to be a 'no' James. Just look at the polling data. Are you seriously suggesting there was a realistic chance of a 'yes' vote?

      Even if a small and fairly consistent yes lead had been established by September (say 52-48), I still think that, on the day, the noes would have it. This is due to the 'bottler' effect I spoke of earlier.

      So in a situation where no is leading in pretty much every poll, there really is no chance of independence.

      Delete
    3. "Are you seriously suggesting there was a realistic chance of a 'yes' vote? "

      Yes, Aldo. Yes I am. That is exactly what I am suggesting. It's always slow progress getting these points across to you, but we'll get there in the end.

      Delete
    4. "..still a comfortable 'no'.
      And that is why there will not be a repeat until the 2020s, at the earliest."

      I'm geniunely puzzled by your cocksureness. Over last few years, support for No (i mean a straight No, as opposed to excluding Dont know) has been treading water around the 50% mark. There is no hard No majority. Obviously if we had a referendum today, odds on No wins, but it now only takes a remarkably small percentage of people to change their minds over the next few months or years for the situation to turn completely on its head.

      If I was seeking reasons to feel confident in the union, opinion polls are the last place I'd look.

      Delete
    5. He's not cocksure. He's trolling.

      Delete
    6. Aldo isn't quite sure whether wants to debate or troll, mainly the latter. He's basically immature, and right now the talk of the referendum result has brought him out of his hide, as it has sentimental value for him in a political climate which is very threatening for him.

      Delete
    7. If it was threatening, it's becoming less so.

      Between where we are at present and Scottish independence, there are four barriers:

      1) The SNP needs to put indyref 2 in their 2016 manifesto. All indications are they will tie it to Brexit. In other words, it will not happen.

      2) A majority of MSPs need to be elected on a pro referendum manifesto.

      3) The British government needs to permit a rerun (very doubtful as we just voted no recently and Cameron has enough on his plate - and his legacy to think about).

      4) "Yes" needs to win the referendum itself. Ahem....no chance!

      So this 'quadruple lock' is enough to keep me happy for now. Consider also that the attack of the Corbynistas, the new devolved powers over taxation, internal bickering, distance from the referendum and duration of time spent in office are all likely to conspire in the next few years to dampen SNP support, then I don't think we unionists have awfully much to worry about.

      It's like being on a plane - you don't start panicking until alarms and flashing lights go off and a mask drops in front of your face. At the moment, the flight is smooth and the drinks trolley is making its way around.

      For the nats, they're still wondering if their flight will ever arrive.

      Delete
  9. @Chan August 14, 2015 at 7:34 PM

    'We really need to stop throwing comments about the margin for error out there without mentioning confidence levels.'

    But we also need to be very careful about what we say about 'confidence levels' otherwise we will be engaging in con tricks. If we have a poll of 1000 plus which yields a result of e.g. 55% plus or minus 3% saying they intend voting no, with confidence level 95% what the mathematics tells us is that IF IT IS THE CASE THAT 55% OF THE POPULATION SAY THEY WOULD VOTE NO THEN 95% of samples of the population of that size would yield no percentages of between 52%-58%.

    A great many people interpret this to mean that IF A POLL OF 1000+ YIELDS 55% NO then it is 95% probable that a poll of the entire population would yield a No result of between 52-58%. This claim is in an important sense a converse claim (as indicated by the capitalised clauses) of the earlier one. There is absolutely no logical or mathematical reason to believe this probability claim (though of course that raises an interesting question as to why polls are usually approximately right).

    The more careful polling companies will often put a clause in the small print saying in effect that confidence intervals should not be interpreted as probabilities (though the rationale they often give for this is arguably confused or at any rate highly contentious); but, like most small print, hardly anybody reads it and I fear that most people on sites like this wrongly treat polling results as being 95% probably correct.

    ReplyDelete
  10. Read "London calling how the bbc stole the referendum" just finished it a very good read on the "Impartial" BBC

    ReplyDelete
  11. The BBC very nearly did steal the referendum - but not in the way you think. Their hosting of the 2nd debate was scandalously biased towards Salmond and the Yes campaign, with Alistair Darling shouted down, interrupted and drowned out at every opportunity. After that 'debate', the Yes campaign achieved a legitimacy and momentum it had not enjoyed beforehand - carrying it to within 5% of victory.

    So when I hear yessers moan about the BBC, I find it laughable. The BBC, if anything, assists you - as could be expected of a left wing organisation.

    ReplyDelete
    Replies
    1. Out of interest, which 'high profile' unionist politicians would defend the union (and drive people towards Yes as occurred last time) in a new iref?

      Delete
    2. Darling and Brown will still be around for quite a while, I imagine. Ruth Davidson is well liked despite being a tory. Kezia Dugdale is articulate and easy on the eye.

      You''re kind of missing the point though - a stuffed toy could front 'no'. It would still win.

      People weren't voting on presentation but on their futures. What will happen to the pound in my pocket? Will the NHS be sufficiently funded? Will I still have a job? Will my benefits / pension / wages retain their previous value? What about debts? Property values? Freedom of movement?

      People looked at the entire proposition and thought "how about NOOOOO!!!"

      I'll front the no campaign next time. We'll still win. Spin becomes irrelevant when you're facing an existential threat.

      Delete
  12. Will the proof be in the pudding as they say? We have our election in just 9 months time and local council elections are showing major swings to the SNP. I know not all SNP supporters, support Independence, but I suspect the huge swathes of people joining after the referendum were joining because they want Scotland to have self determination.

    The support for SNP should at least in part reflect what the people really want for their country to move forward into the 21st century, and not back to victorian times, which is the way westmonster are taking things.

    Let's see what happens in the coming months and do all we can to counter project fear #2, the establishment will stop at nothing to keep us shackled. Let's work to throw off the chains! Thanks to all who continue to reveal the lies from the msm, where would we be without that.

    ReplyDelete
    Replies
    1. There's nothing Victorian about attempting to live within your means and shrinking a massively bloated welfare state that doesn't even do what it says on the tin any more.

      Only socialist dreamers think we can do it differently.

      Delete
  13. pedant alert

    abley supported by a bunch of bt trolls,

    i dont trust you kelly, nor will i ever crowd fund you again

    SC

    ReplyDelete
    Replies
    1. You're the troll, mate. Your money isn't wanted or needed, so sling your hook.

      Delete
    2. Sometimes I forget that nationalists aren't some monolithic threat like the Borg in Star Trek, but actual human beings who squabble and fall out with each other as well as with the other side.

      Delete
    3. Well, that's generously assuming that "SC" is actually a nationalist. When I last ran a fundraiser in autumn of last year, it was extremely obvious that a concern trolling operation was going on - unionists posing as nationalists saying things like "sorry James, but I'm not going to fund your living expenses, I always thought this blog was your personal contribution".

      But they were wasting their time trying to sabotage it - the target was exceeded by almost £1000.

      Delete