Sunday, September 7, 2014

A wander through the YouGov and Panelbase datasets

I've now had a look at the datasets from both the YouGov and Panelbase polls - the Panelbase datasets aren't on the firm's website yet, but Ivor Knox very kindly sent me a copy.  Of course the first thing I always look for is the unrounded voting intention figures (or rather the figures rounded to only one decimal place), and on that front there is good news and bad news...

YouGov :

Yes 51.2% (+4.4)
No 48.8% (-4.4)

That's better than I originally expected, and the reason is that the YouGov poll was actually slightly misreported in many media outlets - the split with Don't Knows included is Yes 47% (+5), No 45% (-3), rather than Yes 47%, No 46%.  As a result, I've had to make a slight correction to the latest Poll of Polls update - the average No lead is now a further 0.2% smaller.

The less good news is that Yes only barely made it to 48% in the Panelbase figures, thanks to the effect of rounding -

Panelbase :

Yes 47.5% (-0.1)
No 52.5% (+0.1)

So that further deepens the mystery of why the Panelbase numbers don't appear to be budging at all (albeit from an all-time high position for Yes), at exactly the same time as the No vote appears to be in freefall with the formerly No-friendly firm YouGov.  The gender gap has been more or less wiped out in the Panelbase poll - No lead by 53.2% to 46.8% among women and by 51.7% to 48.3% among men, which reverses a modest Yes lead among men in the last poll.  It would be tempting to believe that the swing to Yes among women must be real and that the swing to No among men is an illusion, but there's a danger of wishful thinking in that kind of speculation - it's just as possible that normal sampling variation is at play, and that the Yes figure is a bit too high among women and a bit too low among men.

Nevertheless, it's true that if you isolate out the figures for women only, today's two polls are actually identical, because YouGov are also showing figures of Yes 47%, No 53% for female respondents.  Moreover, YouGov concur that there has been a narrowing of the gender gap - the swing to Yes among women since Tuesday's poll is a full 5%, while among men it's only 3%.  But even though two polls are suggesting that the female Yes vote is closing the differential, that is still far from being absolute proof that it's really happening.

One thing that is really striking about a comparison of the two polls is that on the raw unweighted numbers, the Yes vote remains higher with Panelbase than with YouGov - but Panelbase's weighting procedures have harmed Yes, while YouGov's weighting procedures have harmed No.  Of course to some extent that is for very good reasons - YouGov's raw sample significantly under-represents lower income voters, who are more likely to be in the Yes column.  But if I was going to raise a question mark, it would once again be over Panelbase's recent introduction of weighting by recalled European election vote, which no other firm is doing and which significantly harms Yes - the 278 people who recall voting SNP in May have been downweighted to count as just 221 people.

By the same token, eyebrows might be raised at the fact that YouGov's weighting by Holyrood vote from 2011 has helped Yes - the 331 SNP voters in the raw sample have been upweighted to count as 375 people, while Labour voters have been significantly downweighted.  But the crucial difference is that YouGov know for a fact that they have an in-built problem in their panel with having too many Labour voters from 2011, because they collected detailed information at the time, and a large chunk of their current panel were already with them back then.

Panelbase and YouGov now have something important in common - they're the only two pollsters so far who have taken the sensible step of introducing weighting by country of birth, which we know is a strong predictor of referendum vote.  Both pollsters are now showing a clear lead for Yes among Scottish-born respondents, although as you'd expect the lead is slightly bigger with YouGov (54% to 46%, compared to Panelbase's 52% to 48%).  And of course in both cases, those respondents have had to be upweighted while English-born respondents have had to be downweighted.  If all six pollsters were doing this, it's reasonable to imagine that the average No lead on the Poll of Polls would be a touch lower.

Paradoxically, though, what's really setting the YouGov poll apart from Panelbase is the findings among respondents who weren't born in Scotland - YouGov say that 32% of people from other parts of the UK, and 46% of people from outside the UK, are now planning to vote Yes.  Panelbase don't have exactly equivalent figures, but they're saying that only 26% of English-born respondents and 24% of respondents born outside both Scotland and England are Yes voters.  If those figures are underestimates (and for what little it's worth, my gut feeling is that they are), that could partly explain why Panelbase are failing to detect the swing reported by YouGov.

Indeed, what's really encouraging about the YouGov poll is that, almost right across the board, demographic groups that have hitherto resisted the Yes message are swinging in our direction.  It's not just women and people born outside Scotland - the Yes vote among over-60s has increased from 31% in Tuesday's poll to 38% today.

Irritatingly, YouGov have departed from their previous practice of showing turnout-filtered voting intention figures in their datasets.  However, the likelihood to vote looks almost identical among Yes voters and No voters, so it seems reasonable to suppose that Yes have exactly the same 51% to 49% lead among definite voters (which would mean that the swing to Yes since Tuesday's poll is 5% among definite voters, 1% higher than among the whole sample).  By contrast, Panelbase's turnout filter is helping Yes - without it the No lead with DKs excluded would be 2% higher.

Final thought : fresh from his red herring yesterday about "expectation management", Kenny Farquharson is now trying to set another hare running about how Yes supposedly need more than 51% in the polls to have a chance of winning, because of the large number of postal votes that were cast when No were still clearly in the lead. Not to put too fine a point on it, that's utter garbage.  If you've already voted, and a pollster asks you how you intend to vote, you're highly likely to tell them how you actually voted.  That means that it's a touch harder for either side to achieve a swing in the polls, but that any Yes lead still means exactly what it says.

When I pointed that out, Kenny said : "That assumes people with postal votes don't change their mind."  Well, no it doesn't actually, but even if we assume for the sake of argument that some people are telling pollsters they plan to vote Yes when they have in fact already voted No (!), that is nowhere near the disadvantage for Yes that Kenny seems to think it is.  Nobody voted by post until after the second debate, and the first two post-debate polls had Yes at 47%.  If the postal votes reflect that position, and we assume that they account for roughly 1 in 5 of all votes cast, then Yes would only need to be on 51% of the vote by polling day to win. (Do the sums yourself if you don't believe me!)

35 comments:

  1. James, what was the % Scottish born in the unweighted base?

    ReplyDelete
  2. SS : With Panelbase, it was 77.4% (weighted up to 83.0%).

    ReplyDelete
  3. Thanks, it's just there's a nice correlation between the number of Scots agreeing to take the poll and a higher Yes lead. This one is a bit lower in terms of Scots agreeing to take it than the last one, with would explain why Yes didn't gain much.

    This problem I don't doubt relates to the 10% who 'flatly refuse to be polled' ICM's Martin Boon mentioned. Of course that won't show up as 10% too few Scots born, but will show up as too few Scots taking the poll at some level, like here.

    ReplyDelete
  4. James - one thing that has struck me as odd about the YouGov datasets is the fact that there is no regional breakdown. I assume that Kellner is weighting by region. But if he's not, could that be one reason why YouGov appears so out of sync with the other pollsters - i.e. a big shift in, say, Glasgow might not be replicated in the Lothians?

    ReplyDelete
  5. Postal votes do worry me, but there is nothing to be done about them now. Just have to make sure the turnout on the day is large enough to negate any No bounce from the postals.

    ReplyDelete
  6. The mystery of why Yes released their Panelbase poll, which James goes into in some detail in his post, is fascinating.

    But, when you reflect on it from the perspective of a political campaign I think it becomes clear

    We can be sure now, can't we, that the intermediate unreleased Panelbase poll did not show a Yes lead and was surely either the same as or less Yes than this new one? That much has been tacitly admitted. With the enormous build-up of expectation following the last YouGov poll, and speculation and rumour swirling last week of a pro Yes poll, any release of such a poll marking time would have seemed dispiriting and anti-climactic.

    However once YouGov dropped their bombshell yesterday with Yes in the (notional) lead, the situation changed dramatically. A poll like the YouGov one is such a shock to the body politic it almost feels psychologically (if not mathematically) as if a Yes win is inevitable. With it out there, there may well be a major financial shock tomorrow Monday , the kind of panic, which Yes will perhaps not greatly welcome. So, as soon as YouGov dropped its bombshell there was no harm at all to YES in releasing their own poll to calm things down and also make it clear the thing was not in the bag. Indeed Nicola Sturgeon made just such a calming speech yesterday, pointing out it was just one poll and seeking to temper expectations.

    Meanwhile all the people on this site who expected and predicted a late spurt for Yes have been proved right. It is however note a much later spurt than 2011. where a very similar constituency poll (40 SNP 39 Labour) was published by YouGov as early as 38 DAYS before polling. A week alter in 2011 Panelbase had the parties level 31 days before the poll and then the big SNP lead opened up 5 days after that, 26 days before the poll. In other words, far earlier than now though, when it came, the shift from level-pegging to a big lead took only 12 days. If polling history repeats itself this looks like being incredibly tight

    BUT if the YouGov collapse of No is fully verified by other polls, then I think it's close to being all over.

    But will it be? I always think 'just one more poll' will show us. But the trouble is that seems to go on and on!

    ReplyDelete
  7. Yes Agree with Expat that there is concerted effort from within the Yes campaign to play down any euphoria, with Alex Salmond even telling Gordon Brewer that Yes were still underdogs and sighting the Panelbase poll as evidence.

    The reaction of BT to the latest YouGov poll will have done a lot of damage to the No campaign, with Ed Millibands extraordinary threat to put up border posts, with the Labour Party in Scotland later denying he said it, doing even more damage as several MSM outlets reported the story, with the slant that he certainly did say it.

    The panicked Osborne offer with equally car-crash tv interview with Carmichael, then the stuttering stammering Darling interview all contradicting each other..

    You can imagine most Don't Knows as well as a lot of soft No's will be watching all this unfold and ask themselves, do I really want to vote for these buffoons!!!

    As for the pollsters getting it right, I think for me the crucial thing is that YouGov have been the most No friendly pollster and because of this kelner himself said they went back over their previous data and compared the people who told them they would be voting No or DK in past polls with what they were saying now, Kellner confirmed that people had indeed switched in the numbers necessary to convince YouGov that their past two polls are an accurate reflection as of what's happening in Scotland.

    If you look at the UK Governments reaction, it would seem they think the YouGov poll is more reflective than the Panelbase.

    Interesting to see how the markets react tomorrow and how the UK Gov react to them.

    Pop-corn at the ready!

    ReplyDelete
  8. So did Kenny Farquharson explain what he was on about when he ominously said that it looked like Yes had "seriously mismanaged" expectations? I thought he could only have been referring to an upcoming poll showing a reversal for Yes. Did he mean the Panelbase, or what?

    ReplyDelete
  9. He meant panelbase. They released the numbers due to the women vote increasing.

    ReplyDelete
  10. Mike Smithson is tweeting that No is comfortably ahead on YouGov unweighted numbers. He knows better so what is he up to?

    ReplyDelete
  11. Fraser Nelson on Sky News today was at pains to point out (stated it twice in one interview) that the Yes lead would grow. It was very intriguing to watch.

    I hear the Sun will out for Yes this week?

    Rupert Murdoch claims he has tweeted fugures of Yes internal polling.

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. I sense that something very strange is happening here with YouGov. The new TNS and Survation polls should help make the picture clearer.

    ReplyDelete
  14. Chris Knowles said...
    "Rupert Murdoch just tweeted that Yes internal pols showing 54Y 46N.

    Not sure what to make of this as old rupert is a very shady character."

    You bet. He will make his play next week. He thinks he's winning. He may well be.

    DENISE though on different sides I am delighted you agree with me I may be a NO but I think I would be YES if it freed Scotland from the scourge of Murdoch. Absolutely clearly coming as I have predicted on this site for weeks. He lost England now he wants you. (At least as far as he can neutrlalise his opponents in England)

    ALASTAIR, intriguing but I don't think Murdoch in formal mode at Leveson is remotely useful. That is pure PR. He continues to tweet his intentions clearly.

    MICK I hear you and we are on the same side in one respect but look at his tweets. The new ones!

    Oh and Alastair. If this turns out to be big, as it probably already is, I think your idea of a peace-making (?) 'dram' after the event for old SGP sparring partners is not such a bad
    idea. Hell, bridges will need to be built

    ReplyDelete
  15. Expat, I think Murdoch is only interested in Scotland to the extent to which a Yes will cause consternation in London. That is where his interest lies and will continue to lie. If he gets his revenge I think it's the last we'll hear of him in relation to events in Scotland.

    ReplyDelete
  16. Phil : As far as I know YouGov aren't weighting by region. But there may be upsides to that as well as downsides - the effect of Survation's regional weighting seems completely random at times.

    Denise : Smithson is right, of course, but if he hadn't petulantly banned me from his site for life, what I'd be asking him is "so what?" It would be very peculiar if YouGov weren't correcting for the under-representation of low income people in their raw data.

    Calum : Survation will tell us a lot, but I'm not sure TNS will be of much use, simply because their fieldwork will be so out of date. If YouGov are to be believed, a third of the 12% swing happened within the last week.

    ReplyDelete
  17. Another theory: The YouGov poll is an attempt to cause the £ to go into a temporary freefall aimed at terrifying the electorate. Another poll will then be released showing we have all taken fright and are voting no again. £ stabilises. No secures victory.

    Just another theory, not saying that I believe it, but tbh wouldn't put anything past them.

    ReplyDelete
  18. Expat - Murdoch's psychohistory is too complex to explore in a format like this, but the Calvinist backdrop, to which Murdoch alluded in what I quoted from Levenson (in comments to the previous item on this website that you have picked up on), is possibly significant in understanding his ambivalent relationship to the Scottish nation and the British state. Too complex to go into here, but you'll find threads of it in one of my Bella articles from a few weeks back - http://goo.gl/yec5pN .

    Meanwhile, as the £ is apparently wobbling as the markets open, let me given a discussion touched on here (I think it was) a few days ago, share a post I just made to Craig Murray's reflection on what he's been picking up from Scot Goes Pop this past 24 hrs. Murray concludes: "There is at last some understanding that Yes will win: the penny has not yet dropped that this is a revolutionary moment, not a polite constitutional shuffle." His piece is at http://goo.gl/TNdjnV .

    My comment, slightly tangential but triggered by his focus on the failure of elite perceptions:

    "From speaking at both a predominantly No and a Yes event this last week I’ve been struck by how much the currency has become a fetish (in the anthropological sense of something taking on a symbolical value beyond actual value) by an elite social class that signally fails to grasp what is going on. Currency is also being used as a weapon – but a boomerang as illustrated by the pound’s fall in today’s markets. The solution is simple. Don’t mess around with offering us all powers except defence and foreign affairs. It’s too late for that, and it only reveals the extent to which this is a bid to hold on to an imperial past that avoids the need to examine identit(ies) in the rUK. Instead, stabilise the markets by rapidly agreeing continued currency sharing in all but name and with due safeguards for the rUK economy. Signal the intention, if Scotland votes Yes, to bring about a just amicable settlement, one in which Scotland will willingly shoulder a fair share of the debt."

    ReplyDelete
  19. Pantone300 - as I was posting the above you nipped in and put your finger on what was in my mind, but had not quite brought to articulation. Well said. Mind you good folks - both Yessers and Noers alike in this discussion - we'd better all be aware of the danger of getting sucked into conspiracy theories over this next 10 days. Recall Kipling (whose mother was one of the MacDonald sisters):

    If you can keep your head when all about you
    Are losing theirs and blaming it on you,
    If you can trust yourself when all men doubt you,
    But make allowance for their doubting too;

    ReplyDelete
  20. "but if he hadn't petulantly banned me from his site for life"

    You would have wasted a great deal of your valuable time on a westmisnter bubble site dominated by out of touch tory twits, bigots and racists who have been proved wrong about the independence referendum time after time after time.

    Amazing weekend for campaigning with the icing on the cake being the No campaign and the westminster media going into complete meltdown and panic mode.

    Again, that outright panic was predicted on here multiple times and so it has proved.

    The spectacle of the BBC and Sky News trying to 'report' the latest developments has been absolutely hilarious. They all look shellshocked, utterly bewildered with some of them looking near to tears. Apart from revealing just how 'impartial' they truly are, you just know they were relying on some of the usual suspects like Curtice and Kellner who previously insisted that there was nothing to worry about. Not any more though. LOL

    Not once did it occur to them that the precise same thing happened in 2011 with the same out of touch pundits, pollsters and westminster media finding themselves completely at odds with the reality on the ground and having to eat their words and staggering complacency as the landslide took hold.

    The simply do not get it.

    The westminster media are all but an irrelevance to the Independence referendum. The levels of trust in them are even more laughable than the levels of trust scotland holds for Cameron, Clegg and little Ed.

    Nor will the polling do anything other belatedly try to catch up to what's happening on the ground just like in 2011. (where it never fully did even at the end)

    These last 10 days will see the fruition and culmination of a broad based and diverse grass-roots Yes campaign that has been YEARS in the making. Everything is in place and we are utterly determined to put the case for a better scotland directly to the scottish people. Friends, family, work colleagues, young and old, rich and poor, Everyone knows a Yes or a No in scotland so fatuous and witless scaremongering was NEVER going to work. We simply do not need the westminster dominated media/press and never did. Why on earth would we bother with a distrusted 'middleman' when we can talk directly to scots across the country? Something we have been doing for years in ever more huge numbers in the past six months.

    Meanwhile the panicking No campaign is sending for the westminster 'cavalry' MPs to save the No campaigning by popping to scotland and lecturing scots on why we are better together. This after securing their 10% pay rise.

    ROFL

    Truly unspoofable, and yes, we predicted this as well because, you guessed it, the precise same thing happened in 2011 as the panic set in and the idiots from westminster sent their most comical 'helpers' to scotland to 'turn everything around'.
    Didn't quite work out that was though, did it? ;-)

    ReplyDelete
  21. "but I think I would be YES if it freed Scotland from the scourge of Murdoch."

    Hahahahahahahahahahahahaaa!!!


    Stop wasting everyone's time as your feeble attempts to troll this site have become frankly pitiful. You clearly have gone round the bend if you expect anyone here to seriously believe that hysterical shrieking nonsense.

    ReplyDelete
  22. Just looking at what's happening on the pound now the London markets are open. In round figures, it's down about a cent on the dollar over the past 24 hrs and between 3 and 4 cents over the past month, with analysts suggesting the Scottish factor as the main driver. However, that is against the backdrop of a steady climb since the end of 2013. This past month has seen it lose only about 2/5 of that climb. As such, the constitutional uncertainty could be seen as doing the UK a favour to limit over-valuation and effect on exports. In consequence, unless we see a much sharper fall then contrary to what I suggested in above posts, the markets may not be that strong a driver towards currency union. They'll already have factored much of the uncertainty in: the bookies certainly have, with today's Oddschecker for several of them falling below 2:1 on yes (down from about 6:1 not long ago).

    On Murdoch, if he does come out for Yes this week, let's remember that the polls preceded his tweets, so don't let him run off with the prize. They who live by Murdoch die by him. Scotland, take note.

    ReplyDelete
  23. this poll will fire No voters out of bed on the day, that's for sure :-)

    237 hours to go:-)

    ReplyDelete
  24. If Murdoch helps win the vote for 'Yes' then everybody on the 'Yes' side should be overjoyed. Nothing could in any way take the shine of a 'Yes' vote.
    It isn't a pact with the devil after all.
    Grow a set and celebrate when the morning of the 19th dawns and we are heading for Independence.

    ReplyDelete
  25. saynotoyesmen,

    my perception is that this poll has demoralised many No voters, rather than fighting back they are just shrugging shoulders?

    also the panic and infighting inside BT is allowing media and soft No's to turn against them

    ReplyDelete
  26. saynotoyesmen,

    my perception is that this poll has demoralised many No voters, rather than fighting back they are just shrugging shoulders?

    also the panic and infighting inside BT is allowing media and soft No's to turn against them

    ReplyDelete
  27. James, how does your PoP look inc DKs? I was enjoying following it as it dipped from > N+20 to ducking under N+10.

    ReplyDelete
  28. Alastair writes "They who live by Murdoch die by him. Scotland, take note."

    Well said and you might note that the Scottish Sun today mocks Better Together as the Sun prepares to bow to its master and endorse Yes.

    I also though agree with the contributor who said Murdoch is primarily in this to attack his enemies in England (which is currently just about the whole English political class apart from Nigel Farage). True enough. But that does not mean he won't expect a reward and influence. He always does.

    I do have to correct my statement though that Salmond was Murdoch's only ally left. I was wrong, Farage is a critical ally of Murdoch's too. And Farage certainly benefits from Yes (even if he is notionally No) for a Yes win is likely to neutralise Labour in England. Note too Murdoch is a passionate opponent of the EU so don't expect him to encourage any European aspirations in Scotland.

    ReplyDelete
    Replies
    1. Per PB, TNS are publishing tonight, rather than tomorrow as originally planned.

      Delete
  29. The last TNS poll had Yes: 45% No: 55% excluding don't knows and unlikely voters. The fieldwork will be out of date so may have missed a very recent shift to yes, but it should be interesting anyway.

    ReplyDelete
  30. Re Panelbase and the 278 Y weighted down to 221 Y because of EU2014 vote 'remembrance'.
    On a rough calculation of say 1000 polled, and putting these votes back in as unweighted, the 475 Y becomes 532 Y, ie 47.5% Y goes to 53.2% Y.

    Interesting: even if not entirely accurate.

    And see graphs of ICM polls pre 1979 Devo Ref.

    https://twitter.com/JamesCo77979225/status/490489074021851137/photo/1

    ReplyDelete
  31. Brian:

    I've been keeping track of the poll-of-polls updates as James has published them. I think I've got them all in this plot: https://dl.dropboxusercontent.com/u/18361224/nolead-sep8.png

    A surprisingly linear trend. Recent points have been flattened by the presence of some much older polls in the PoP, hopefully that will all change of the next few days.

    ReplyDelete
  32. James Coleman - That isn't the first time I have heard such a number.

    Will be even more interesting once we find out the numbers of new voters/lapsed new voters as well.

    FT.com is going mental, George Robertson has an article up, horrible stuff.

    ReplyDelete
  33. Thanks Sandy. Tonight's TNS might provide some welcome relief for Prof Curtice's PoP but likely also to boost ScotGoesPop. Everyone's a winner. ;-)

    ReplyDelete