Friday, September 5, 2014

A few thoughts on the 'missing million' conundrum

On the last thread, Chalks pointed to the doubts cast by John Curtice on the claims from the Yes campaign that their pursuit of the votes of the "missing million" means they are picking up support which is being missed by the conventional opinion polls.  Curtice bases his scepticism on an analysis of the voting intentions of poll respondents who say they didn't vote in the 2011 election, and who he claims are actually shown to be more likely than others to be No voters.  Straight away, that sets a number of alarm bells ringing in my head, so here is your cut-out-and-keep-guide to why you should at least maintain a healthy scepticism about Curtice's scepticism...

Four out of six of the active pollsters in this campaign conduct their fieldwork among volunteer online panels.  One point I made in the interview for the Phantom Power film a few weeks ago (and which didn't make the final cut) is that to the extent that online pollsters have proved their credibility in recent years, you could easily make the old joke : "I know it works in practice, but does it work in theory?"  Polling among volunteer panels shouldn't work in theory, so the firms in question don't even worry about that - but they do go to great lengths to make sure their results are as accurate as possible in practice by 'working backwards'.  If a general election has just happened and a pollster's findings weren't quite right, they ask themselves how they could, for example, tweak their weighting procedures so that their raw data would fit the actual result.  If they also have evidence that the tweaked methodology would have produced reasonably accurate results in previous elections as well, then bingo, they've got a refined methodology that works in practice and will probably work in future elections as well.

But that's where they may be coming slightly unstuck with this referendum, because they don't have any baseline to work from to test that they are getting their methodology right - there have been no independence referendums before, and we know this contest is going to be radically different from normal elections, partly because of higher turnout and higher levels of voter registration.  That doesn't necessarily mean any given pollster is bound to be getting it wrong, but it does increase the degree of uncertainty (hence the unusual amount of variation between different firms' findings), and the biggest area of uncertainty for any online pollster will be people who usually don't vote, and people who haven't previously been registered to vote.  There simply aren't enough of those people "on the books" of volunteer online panels - in normal circumstances there don't need to be, because online pollsters only need to be right in practice, and in practice the 'missing million' don't count at all in normal elections.

So we should certainly doubt any analysis of the voting intentions of previous non-voters that draws too heavily on the findings of online firms.  There is also room for doubt with Ipsos-Mori, who by definition are only reaching people who are willing or able to answer a landline telephone, meaning there is no way of knowing whether enough of the 'missing million' are being contacted.  The one firm who in principle should be delivering the goods is TNS-BMRB, who actually go out into the real world and knock on people's doors - so they ought to be reaching people in the most deprived communities as easily as they are reaching John McTernan's butler.  And, unfortunately, it's true that TNS-BMRB have tended to be one of the more No-friendly pollsters (albeit usually not quite as No-friendly as Ipsos-Mori or YouGov).  But it also has to be borne in mind that their numbers go through a very unusual weighting procedure.  For example, here's what happened in the TNS poll conducted in June -

Only 226 people were found who said they didn't vote in 2011, but they were upweighted to count as 320 people.

124 people were found who couldn't remember how they voted in 2011, but they were upweighted to count as 173 people.

As far as I can see, the logic for upweighting both groups is the same - TNS seem to think that all of these respondents are representative of non-voters from 2011 (and indeed of previously unregistered voters).  Quite why people who don't recall how they voted should be automatically treated as abstainers is a bit of a mystery, and that's the first red flag we need to raise about the TNS approach, because in most cases it's the upweighting of the "forgetters" that is actually helping No the most.  If we assume for the sake of argument that many of these people did in fact vote in 2011 but genuinely can't remember how, then that significantly changes the picture that the TNS data is providing about the 'missing million'.

But the broader question is why such sharp upweighting needs to be happening at all?  I think there are two factors at play here.  Firstly, there must be a lower response rate to TNS polls among people who don't usually vote (ie. those people either won't answer their door, or will be more likely to turn the interviewer away).  So that introduces at least a degree of the same uncertainty that applies to the other pollsters.  The second factor is one that Professor Curtice hasn't even acknowledged, as far as I can see - it's likely that a significant minority of people are lying, and are telling TNS they voted in 2011 when they didn't, largely out of embarrassment.  Those people will presumably give as their "vote recall" the party they would have voted for if they'd actually made it to the polling station, or perhaps the party that in retrospect they'd like to think they would have voted for.  So there will actually be members of the 'missing million' who are telling TNS they voted SNP in 2011, and who are being weighted accordingly.

None of this is to say that a high turnout and a high level of voter registration is bound to favour Yes - but I do think Curtice's specific objections to that notion are not based on particularly solid ground.

*  *  *

A small correction to yesterday's post : Ivor Knox of Panelbase sent me an email earlier today to clarify that all of his firm's polls for the Sunday Times have been commissioned by the Scottish edition of the paper, whereas the upcoming YouGov poll has been commissioned by the UK edition.  So Panelbase haven't been sidelined, although Mr Knox stressed that he couldn't say whether they'd be conducting any further referendum polls for the Sunday Times.

That still leaves the mystery of who commissioned the Panelbase poll that has been in the field this week - perhaps we'll find out over the next couple of days.

*  *  *

This may not be an entirely original observation coming from me, but Political Betting's editor Mike Smithson hasn't exactly been covering himself in glory of late.  I've just spotted this tweet, which refers to a comment from Ivor Knox about protecting the client's right to confidentiality -

"My reading of @PanelbaseMD Tweet is that he has new IndyRef poll which client doesn't want to publish #BadforYES?"

If Mr Smithson had been paying attention, he'd know it was established beyond reasonable doubt several days ago that there was an unpublished Panelbase poll conducted last week, probably for the Yes campaign.  That may mean that Yes failed to get a significant further boost in that poll, but that's pure speculation, and even if it did happen to be true, the fieldwork ended three days earlier than in the breakthrough YouGov poll.  There has also been a second Panelbase poll in the field this week, but I know of no evidence that it has been withheld - indeed it may not even be completed yet.  I first heard about it on Wednesday.

UPDATE : There's confirmation from Teri in the comments section below that the fieldwork for this week's Panelbase poll didn't conclude until today (Friday), so Smithson's claim about a "new" poll being withheld is complete and utter nonsense.

And a special message for Neil Edward Lovatt (for I know you're reading this) : please stop using this blog as a bogus "source" for your eccentric rumour-mongering on Twitter.  You're a disgrace to "risk assessors" everywhere.


  1. I hear that the over 7,000 registered to vote in Dundee in the past few weeks so that would be roughly about a 6 to 7% increase in electorate. If these people do vote and are mostly Yes voters then it could make a 4 to 5% difference?

  2. Dundee is a YES stronghold, you can't doubt it whenever you go there just now. The place is practically bouncing, and there are yes flags, stickers and posters everywhere you go. Unfortunately over the water in Fife it isn't quite as clear cut. I live near Leuchars in archetypal middle class suburbia and have never had a YES canvasser at the door, although we do get leaflets. I have had BT at the door once, it was their local Lib Dem councillor and whilst he seemed a nice chap, I didn't detain him on a wet night as he could never persuade me.

    I think Fife, like a lot of west central Scotland will be a key area, with a lot of variation within the region in terms of voters because of the demographics involved here.

    It would be fascinating to know how new voter registrations align in terms of social background.

    Whether these new voters are in favour or YES or not we have to be inspired by the level of participation and engagement. I suspect what impact they will have is just one more factor to add to the level of excitement. I don't think it can be predicted either way. I want to believe what RIC and others are saying and applaud their efforts, but I know a lot of middle class voters who do not normally vote and who are also now equally engaged. But even at St Andrews university, amongst the staff, which is riddled with Oxbridge types, it is refreshing how many are voting YES, and indeed how many English academics are. The student body is pretty divided I'd say due to the high intake of English private school kids who tend to be very NO oriented, but due to the term dates I am not sure how many (especially of the new students) will be registered to vote here. Many of the international students would love to vote YES and support us, but they don't get a vote unfortunately.

    When are we going to start discussing the regional variations we are expecting in the referendum result? Which counting area will be the first to declare? Imagine the excitement if the exit poll shows it being very close.

    Also, and I am sure this has been asked before, but what happens if it is a draw? This is of course possible, although the very thought of it would be heartbreaking. An interesting point to discuss nevertheless.

    YES FM is well worth a listen for some inspiring music!

  3. "Imagine the excitement if the exit poll shows it being very close."

    I'm a relatively new reader, so I don't know if James has addressed this, but it was my understanding that there are not going to be any national exit polls on the 18th. Scandalous if true, of course.

  4. Anon

    Even within Dundee there are pockets of No support.


    I did read somewhere recently that did conform there will not be any exit polls on the night which was the same as in 1979 and 1997.

  5. I'm glad there will be no exit polls it will be more exciting

  6. "But what happens if it is a draw?"

    Toss a coin? Highest card wins? I believe that is what happens if a constituency election is a dead heat. Would be a great way to end the union.

  7. "Also, and I am sure this has been asked before, but what happens if it is a draw?"

    It hasn't been asked before, because the odds against it happening are astronomical. With elections involving much smaller electorates (like local council elections) it does very occasionally happen, and if memory serves me right the returning officer is allowed to toss a coin or cut cards to decide the result. Crazy but true.

    "Many of the international students would love to vote YES and support us, but they don't get a vote unfortunately."

    Some do - EU and Commonwealth citizens are able to vote (if they registered in time).

  8. Result around 10am 19th Sept.
    Don't know whether to go to bed the night before. Won't be able to sleep anyway.

  9. Actually, the toss of a coin thing happened to my father a number of years ago in Tadcaster (north England). At the time he was standing for the Labour party, and up against an independent. We thought we`d done enough to win the seat on the council, but the result (after 2 recounts) indicated a dead heat. Cant remember the size of the electorate, but it might have been 500 votes each or something!

    Anyways, to cut a long and not so interesting story short, both marked new ballots, put them into a box, and the returning officer drew one name out of the box. Luckily it was the old mans :-) A couple of years later after speaking out against the Iraq war (and having received death threats) he resigned from the Labour party (40 year long membership). Naturally he`s an SNP member now..

  10. There seems to be an unwillingness to apply basic principles of Sociology and Psychology to the increased turnout. Given that there is no statistical basis for the assertions being made by both sides, due to a lack of precedent on which to base the weighting, only socio and psychological estimates can be used.

    Of course this then becomes a judgement call which I am sure psephologists detest.

    My judgement is that are there two cores for new voters. Died in the wool Orange Order/Rangers/Hearts/NIUnionist affiliated individuals and the Disenfranchised.

    Obviously the first category, if they choose to register for the first time will vote NO but the latter will vote Yes because basic sociology says dominant behaviour is not changed to maintain the status quo.

    This has to be highly positive for Yes. Firstly because, despite some potential overlap, having grown up outside West Central Scotland, I can be confident that the Disenfranchised outweighs the "Affiliated".

    I can also be confident that the consistant media narrative that No was miles ahead and it was a "done deal" only left perhaps two days for the "Affiliated" to register while grassroots campaigns like RIC have been motivating the Disenfranchised to register for the last two years.

  11. There should be a distinction between the 'missing million' and the non-voter. There is now anecdotal evidence that many those already on the register but not voted for some time or never voted will do so at this referendum.

  12. James - I love your blog, I rely upon your blog and sometimes it saves my sanity.

    Keep on doing your unique and exceptional stuff.

  13. I agree with Stevie. Up until a couple of months ago my first port of call was WOS. Now it's Scot Goes Pop. So well done James! :D

  14. Thanks james, famous at last! Explained perfectly, curtices theory has no basis or evidence when scrutinised....a bit like another bunch of people

  15. James, this week's Panelbase Poll ran from Monday 1st September and closes today, 5th September. I assume the results may be ready for this Sunday.

    As for last weeks' Panelbase poll, Severin Carrel of the Guradian tweeted during the week that he heard that YES Scotland were sitting on the results of a poll that showed YES in the lead. He said they were waiting for the optimum moment to release it.

    Obviously I don't know if this is true or not.

  16. Teri : Thanks for the information about this week's poll - that pretty much explodes Mr Smithson's theory about a "new" poll being withheld.

    Severin Carrell (I can never remember how to spell his name!) tweeted again shortly afterwards to say that Yes Scotland were flatly denying what he'd just said.

  17. Anyone know if the results are to be declared individually by local authority area during the night or do we have to wait until all votes are counted throughout Scotland before they make one big announcement in Edinburgh?

  18. I'm sorry James. All this nonsense about the missing millions from Curtice, the pollsters AND you strikes me as trying to put an unrealistic validity on the results of the Polls. I do not believe for one minute that all their finagling of figures will validate their polls for the forthcoming vote. Nor do I believe that TNS have ventured into the REAL heartland of the big estates whether doors were opened or closed to them.

    ALL of them are farting in the wind and their results are no better than holding your finger up and saying it will be YES if that is your bent or the opposite if you are a NO person.

  19. I ought to clarify my last post a little. I should have added that the results of canvassing, straw polls in newspapers, polls at public meetings and the "general feeling on the street" all seem to me to be far more realistic about the way the vote is going than any of the "scientific" polls being conducted by polling organisations.

    And I think the YES/NO people agree about that.

  20. James Coleman : Why wouldn't TNS be at least attempting to venture into the real heartlands? Presumably they must have a procedure for randomly generating postcodes to visit, or something like that.

  21. Anon in Fife,

    If there haven't been any yes canvassers in your area, could you become the yes canvasser in your area?

    Anon in Glasgow

  22. By the way, James, you've raised the issue of straw polls in newspapers before - those are useless. They're literally useless.

  23. By the way, James, you've raised the issue of straw polls in newspapers before - those are useless. They're literally useless.

    As an absolute measure Yes.

    However, when you include such 'indicators' in an overall picture they can hint at something.

    The SNP win in 2011 was quite obvious from studies of social media activity well before the pollsters caught on.

    MORI have a short report on it. The main chatter was about the SNP and Labour as you might expect. The mentions for SNP were consistently a tad higher. What really stood out though was that the mentions for the SNP were very positive, and the those for Labour very negative.

    Add the two together and...

  24. James Kelly said...
    James Coleman : Why wouldn't TNS be at least attempting to venture into the real heartlands? Presumably they must have a procedure for randomly generating postcodes to visit, or something like that.

    If you drive about Scotland, or even just Glasgow, you will quite easily find areas you would not be comfortable walking around, even with other people close by. Especially if you are in the socio-demographic that a polling firm will employ as a canvasser.

    Can you really imagine some Yummy Mummy parking her X-3 in Hardgate while canvassing the concrete and roughcast tenements of the area?

  25. I just realised that mentioning Hardgate probably plays into pre-conceptions that this will just be a West of Scotland disease. Replace that with Corstorphine or Raploch and you get the same picture. The demographic of a Polling company canvasser will not park their X-3 in the middle of such urban wastelands to go door to door polling on her own.

  26. Hello James, very busy so a flying visit just to encourage even more of the incredible show of support on this thread for Wee Ginger Dug after his sad loss.

  27. Though I do also just have time to say, easy there guys! It's not quite the wild west it was. Sure there are still some areas where I wold be most surprised to see a great deal of polling firm canvassers but it's hardly all like that by any means. :-)

  28. Anonymous said...
    Anon in Fife,

    If there haven't been any yes canvassers in your area, could you become the yes canvasser in your area?

    Anon in Glasgow

    I have done some canvassing but YES does not bother with middle class areas here, all the efforts are in mostly poorer areas which are the top priority. Needless to say BT, who are in the main wholly anonymous (I have never ever seen a BT activist anywhere in Fife or Dundee) seem to target more affluent areas. To that extent therefore we seem to keep to our separate areas. I have some friends who support BT in Aberdeen and they frequently post facebook updates on their amazing canvass returns, but it is always in the richest parts of Aberdeen. Always makes me chuckle.

  29. Suggestion from Dundee that there could be as many as 180,000 additional voters who have registered in Scotland for the Referendum.

    I've always thought that these additional voters would be overwhelmingly C2DE, but I'm interested in hearing other people's thoughts.

  30. Thanks james, famous at last! Explained perfectly, curtices theory has no basis or evidence when scrutinised....a bit like another bunch of people

  31. Apologies,my phone is a dick.

  32. Apologies,my phone is a dick.

  33. Psephologist Curtice was one of two or three others that used to be wheeled out on a regular basis but that changed not long after 2007. I wonder why?

    We all carry a bias as that is human nature which can be muted if different profs are used on a regular basis keeping each other on best behaviour.

  34. The head of Aberdeen City Council was speaking to my brother last night and he said that they had an unprecedented number of people applying to register to vote just before the deadline.

  35. Re Curtice, he participated some months ago in a panel discussion on the referendum with Vernon Bogdanor, Adam Tomkins and one other, whose name escapes me, at the Royal Academy in London. All of the panellists were unionists, and at the end there was some interaction between them. I think it was Tomkins who asked Curtice whether he thought yes could win and in effect he said no, so he comes to all this with a preference for a no vote, and a pre-existing belief that yes won't win.

  36. I've heard there'll be a Poll in this weekends Sunday Time that shows only 1% gap between the camps

  37. Sad bigots over on Cif still using the Curtice Poll of Polls to prove that No is 10 points ahead. They just can't stop the lies.

    What happens if you take the raw n numbers from Panelbase and apply the ICM algorithm? Or use the Kellener fix? Wouldn't that give a true margin or error instead of the false 3% that gets bandied about all the time?

  38. Interesting Alex, if that is true?
    The Sunday Times poll is being done by Panelbase - where are you hearing this?

    Overall I find it strange that not one over poll this week has been released apart from the YG one. Considering we are 12 days away.

    I can't figure out this possible Yes Scotland previously held back Panelbase poll either.

    It's like Chess now.

    One thing we all know just from looking around and speaking a loud. There is most definitely movement to Yes.

    Still, we'll soon see over the weekend.

  39. There are all sorts of rumours flying around, and after our experience of the last week, I think our default assumption should be that most (if not all) of them are wrong. A couple of days ago Labour troll Duncan Hothersall started a mischievous rumour about the Record being about to publish a poll showing Yes in the lead, and 12 hours later Yes supporters were discussing it among themselves on Twitter as if it was established fact!

  40. The telegraph seem to be expecting good Yes polls tomorrow

  41. Apologies, I mean this new Sunday Times poll is a YouGov one too.

  42. Where does it say that Denise? I had a wee scan through that article earlier, not it all, but all I got from it was a very desperate and paniced Cochrane roaning on about draw a NO on A4 paper and do it for Blighty etc1

  43. "The Sunday Times poll is being done by Panelbase"

    Not this one - it's being done by YouGov, which makes the idea of a 1-point gap seem considerably less plausible. If anything, I would expect that poll to show some kind of reversion to the mean (ie. at least a small bounce back in the No lead). But we'll see soon enough.

  44. Cheers James. I realised when I typed it was YouGov.

  45. Yes it was, as James says, just a rumour. I picked it up on the comments to the article by Mike Small on cif.

  46. From the article : The separatists are in a buoyant mood thanks to opinion polls that seem to suggest – suggest that’s all – that they’re in with a chance of breaking up Britain. And those who oppose them should be aware that there might well be other surveys that suggest the same or worse for the Unionist cause.

    The other surveys worse for the unionist cause

  47. Kevin

    Yes I thought it was strange that there has been no polls this week. Peter Kellner did say on the review of the papers on BBC 24 hour news that he had three more indyref polls, so two more Yogov polls must be following Sunday's poll in relatively quick succession.

  48. Oh, God. That means the Curtice Poll of Polls will shortly be (at least) 50% made up of polls from a single No-friendly company.

  49. @Denise
    And those who oppose them should be aware that there might well be other surveys that suggest the same or worse for the Unionist cause.

    How does that imply that the Telegraph are expecting good Yes polls tomorrow?

  50. I think every dramatic poll result so far was trailed a few hours before by people who would know, such as Nick Robinson with the last YouGov. If that isn't happening, and the rumour is just based on people talking to each other in comments sections, there's no dramatic poll.

    As James notes, some No people are stirring up these rumours as a deliberate tactic, knowing there will be plenty of Yessers willing to propagate them. Let's not help them out.

  51. This comment has been removed by the author.

  52. Cheers Denise,

    Apologies, I missed that bit :( He could well be on about that supposed Panelbase poll.

    Och well, we'll see tomorrow, folks.

    Good point Keaton. We'll know tomorrow at around 10-11 pm just need to check McDougall or Kevin Pringle's Twitter to see if they've re-Tweeted or Tweeted anything.

    Wonder if Darling not agreeing to another debate, to use an American Football analogy, BT are trying to run the clock out, by refusing debate, lack of clear polls.

    In football terms, BT won the first leg 3-1, but now we're at home, we're leading 1-0, there is 10 mins left, they are down to 10 men, our fans are going bezerk, the last 10 mins we've been all over them, all we need is one goal (for that away goal on aggregate 3-3 win) 50.1%

    Anyhoos, nicht for me.. Too much blethers.

  53. Apologies,my phone is a dick.

  54. Blimey, Heffer's article in the DM this morning should be sent to every household in Scotland, a YES avalanche would then be guaranteed!

    There are numerous references in the papers this morning suggesting the NO camp feel more disastrous poll news is on the way this weekend. I hope this is true for momentum reasons, though it has been very amusing the last two days to read the exponential growth in referendum articles being churned out by the London media, much of it all in a panic about what is happening. You'd almost think those sage souls hadn't seen it coming!

  55. @Alasdair

    Re: Orange Rangers supporters.

    You cannot assume such people are automatic No voters. Many organgemen are voting Yes because they are primarily monarchists, not unionists and we are keeping Brenda (well played Eck).

    This referendum cuts across ALL groups, parties and demographics. I have met so many elderly Yes voters and so many women are Yes too and in my experience no more likely to be No.

    You make the statistical error of firstly assuming uniformity then that if the polls say X is bigger than Y then Y cannot exist. Though I'm not aware of any poll that weights by orange or Rangers membership.

    That is what makes this referendum both so exciting and so hard to pick by polling.

    Maybe my sample is biased because I campaign in Dundee (the working class areas are so solidly Yes, but you can still find Noes it is inspiring). But when you meet so many Yes voters from groups supposedly so majority No they can be 'relied on' to vote No it causes you to doubt the polls.

    As others have said, Dundee will vote Yes by a significant majority. You can count on that.

  56. Interesting tweet a couple of hours ago from Kenny Farquharson:

    "There's been some serious mismanagement of expectations by the Yes camp this week, if what I'm hearing is true."

    Maybe news of a possible bad poll for Yes? What I've found interesting about these Panelbase rumours is that many of those spreading them have been Better Together people!

  57. Re that tweet from Kenny, I'd personally be absolutely delighted if things just stayed the same as the polls from last week, nice and close but just behind works for me right until the last moment. We don't need to be winning right now, we just need to win come the 18th.

    Better to be just behind on the bend and win the race by a nose as we race towards the finish line.

    The week of the referendum is the real decision time, but the armies of YES people out and about on our sunny Saturday will be making all the difference. With the media offensive already proved a failure, those enthusiastic boots on the ground are the key to victory.

    #yeslive via livestream is something to accompany the saturday, with things kicking off at 7pm in earnest.

  58. The BT mob are obviously spreading these rumours so they can spin anything less than a Yes lead in the next poll as a setback. It's an attempt to halt the Yes tidal wave.

  59. Very true, Juteman and Anon.

    If things are still as you were give or take 1-2%, I think it's fine and like you say we'll push on for the last week.

    Although if any poll had a Yes lead or 50-50, I think we would see a tidal wave of support from the soft no/don't knows, so if polls show a dip/static for Yes then BT can claim that there's no progress for Yes and it keeps them in the race.

    A friend of mine from Norway was recently in Scotland and his words "If you looked at buildings, it will be a landslide Yes"

    We have a huge grassroots. I know people who are non-political, the "We don't ask who we voted for" types who are taking 3-4 days off to campaign for Yes.
    Yes will be flooded with volunteers for the last few days - getting the vote out.

    No will have to rely on bussing in activists and probably paying them from England, Wales.

  60. From the devil himself.

    Possible further tightening from Yougov?

  61. Yeah that tweet from Murdoch has me very excited, not because I like him, but because it can only be interpreted as good news for YES.

    We need to keep the momentum going. Scotland is waking up before our eyes, it is a beautiful thing to behold. Let's keep it going!

  62. London is going to shit a brick.

  63. London is going to shit a brick.