Another setback for Ruth Davidson as SNP win local by-election in rare Scottish Tory heartland

If Twitter was anything to go by, the Tories threw the kitchen sink at yesterday's East Ayr by-election, and no wonder - even though the SNP were technically defending the seat, the Tories won the popular vote in the ward last time around, so simply by standing still they should have been able to "gain" the seat.

The outcome will delight connoisseurs of the intricacies of the STV system, because the Tories won the popular vote again (albeit only just, after suffering a small net swing to the SNP), but still lost the seat after lower preferences were redistributed.

East Ayr by-election result :

Conservatives 38.5% (+5.5)
SNP 38.0% (+6.5)
Labour 16.2% (-7.3)
Independent 5.5% (-1.0)
Greens 1.9%

I'm rather incongruously spending the anniversary of the referendum at a Union Jack theme park in the east end of Glasgow known as the Davis Cup, but I'll add analysis later when I have a minute...

UPDATE : It may not be apparent on TV, because the cameras are mostly pointing the wrong way, but there's an absolutely enormous Yes flag in the crowd at the Emirates Arena.

UPDATE II : Apologies for the long delay - after leaving the tennis last night my diary proved to be fuller than I expected.

Basically what lost the by-election for the Conservatives was that Green transfers broke heavily for the SNP, and Labour transfers went slightly more for the SNP than the Tories.  There are two ways of looking at that - on the one hand, it illustrates why the hopes of massive unionist tactical voting in May were wildly over-optimistic.  Clearly, when push comes to shove Labour supporters are slightly more interested in voting SNP to keep the Tories out than the other way around.  On the other hand, it's quite striking that very nearly as many people who voted for a party led by Jeremy Corbyn transferred to the Tories as transferred to the SNP.  That suggests there may well be some irrational antipathy towards the SNP among the rump Labour support.

The first thing we always look out for in local by-elections is the net swing from Labour to SNP.  In this case, it was just under 7%, which is very much at the lower end of the range we've been used to recently.  Of course, you'd expect the swing to be lower in a ward where Labour aren't one of the top two parties, but nevertheless it's slightly troubling that the Tory vote increased by almost as much as the SNP's.  The most negative interpretation is that a large chunk of the lost Labour vote went direct to the Tories because of the Corbyn factor, and that the real movement from Labour to SNP was very, very low by recent standards.  A more encouraging interpretation is that the Tory increase mostly came from the independent candidate who didn't stand this time, and that Labour lost their votes en masse to the SNP.  Take your pick.

Corbyn might be deposed before 2020, but he might not be

Keiran Pedley works for the polling organisation GfK NOP, which for many years now has not conducted voting intention polls (at least not for publication), but has had a hand in the broadcasters' exit polls on general election night.  He also presents the podcast Polling Matters, which in spite of its unfortunate association with a certain website is often a fascinating and enlightening listen.  But aside from his expertise when he has his impartial pollster's hat on, he also appears to be a member of the Labour party, and had very strong views about the possibility of Jeremy Corbyn becoming leader and the potential impact on his party's electability.  It's within that context that his assertion that Corbyn "is not going to lead Labour into the 2020 election" should be seen, because the reasons he gives smack of extreme wishful thinking...

1) "He will be 70 going into the general election."  So what?  Many of the potential candidates for the US Presidency are envisaging holding the office at a much older age than 70.  If Joe Biden stands and is elected, he will be 74 by the time he is sworn in.  Ronald Reagan, of course, was almost 78 when he finally left the White House.

2) "It is not even clear he wants to be Prime Minister."  I sometimes wonder if Corbyn's opponents are mistaking his natural modesty and caution for a lack of commitment.  But even if there's any truth in the suggestion that he doesn't want to be PM, it could be just as easily argued that he doesn't want to be Leader of the Opposition - yet here he is.  The reason he put himself forward as leader is the same reason for thinking he'll try to remain in harness for a lengthy period - he feels a sense of duty to his wing of the party, which has been frozen out of influence for so long.  He knows the consequences of stepping aside could be a return to New Labour, a prospect that would fill him with disgust.  He might only be tempted to retire if it looked like, at worst, a soft left candidate (Lisa Nandy's name is sometimes mentioned) would replace him.  To get to that point may require reform of the nomination rules for leadership elections.

3) "His initial poll numbers are dire."  The polling done since Saturday has been extremely limited, but in any case, history is littered with leaders who struggled on to a general election in spite of it being obvious they would lose - Michael Foot in 1983, John Major in 1997, William Hague in 2001.  In many ways, what happened to Iain Duncan Smith was the exception, not the rule.  Even he came amazingly close to surviving in the decisive ballot.

None of this is to say that Corbyn will definitely make it through to 2020 - there are plausible scenarios that could see him deposed, possibly even before Christmas.  But to suggest it's a foregone conclusion that he'll go just seems daft to me.  I also think Pedley is barking up the wrong tree in thinking Alan Johnson is a viable 'unity' candidate who could replace Corbyn.  Labour right-wingers are going to have to come to terms with just how far to the left the membership has shifted - it will now take someone from the soft left to unite the party, not a Blairite with a working-class accent.

Breaking : Survation telephone poll underway

I've been contacted by a reader whose daughter has just been polled by Survation over the phone.  Questions included Yes or No to independence, Holyrood voting intention, and whether Jeremy Corbyn makes you more or less likely to vote Labour.

It's rare for Survation to conduct Scottish polls by phone (they generally use their volunteer online panel), but it's not completely unheard of.  We saw two telephone polls from them immediately before the referendum, and one immediately afterwards.  The first of the pre-referendum polls was commissioned by McDougall HQ, and unusually for a No campaign internal poll, was published - probably because they were panicking about the polling narrative in the media and wanted to calm things down.  The second was an eve-of-polling special for the Record, who in normal circumstances save money by commissioning online polls from Survation.

The new poll could be a private commission from Labour, in which case it'll never see the light of day, but I think it's more likely that a newspaper is splashing out on a telephone poll to mark the anniversary of the referendum.  If so, it's going to be absolutely fascinating to see what the findings are on independence, because this will be the first time we've ever seen the combination of a 'real world' data collection method (which in recent weeks has produced clear Yes leads) and weighting by recalled referendum vote (which more often than not has produced very slender No leads).  I genuinely don't know what to expect.

We'll also get the first meaningful pointer to whether the SNP's massive Scottish Parliament lead has been dented by Corbyn becoming UK Labour leader.

Has the new question transformed the EU referendum campaign?

It's become clear from two online polls in recent days that the new question for the EU referendum has made a huge difference to the state of play, with both ICM and Survation suddenly showing a statistical tie (the former has a narrow 'Remain' lead and the latter has 'Leave' slightly ahead).    As with polling on Scottish independence, there's a gulf between the findings of online and 'real world' pollsters, so it's likely that the next telephone poll will still show a significant lead for 'Remain' - but it probably won't be as big as before.

Anthony Wells has made the point that the impact of the question will disappear by polling day, because by then most people will have made up their minds how they want to vote, and the exact wording on the ballot paper won't sway them.  I think that's broadly right, but in a sense it misses the point.  The real significance of the wording is two-fold -

1) The new question isn't simply the one that will be used, it's also much more neutral and even-handed.  So the polling done with it is likely to provide a more accurate snapshot of public opinion - meaning we may have severely underestimated the popularity of Brexit until now.

2) Polling that is more favourable for 'Leave' will to a large extent shape the course of the campaign.  We saw during the independence referendum that the huge No leads reported by three of the six regular pollsters (including most disturbingly the rather artificial figures generated by YouGov's controversial "Kellner Correction") helped to suck the life out of the campaign until a very late stage, and gave some No-leaning voters an excuse not to think about the issues in too much depth, because they "already knew the result".  A long campaign in which the race looks tight throughout will be a very different proposition.

I do wonder if the polls also played a crucial role in Jeremy Corbyn's election as Labour leader.  In past leadership contests, many natural left-wingers may have been deterred from voting for their dream candidate (for example Diane Abbott five years ago) by the perception that it was a complete waste of time.  Probably union endorsements played the biggest role in making Corbyn look more credible on this occasion, but I suspect the two YouGov polls didn't do any harm either.  It may be significant that the landslide was even bigger than YouGov suggested a month ago.

*  *  *

Stephen Bush of the New Statesman has been vindicated this year by two brilliant predictions - that the Conservatives were on course to do much better at the general election than the polls suggested, and that Corbyn was a shoo-in.  So it's slightly surprising that his analysis of Labour's shambolic whipping operation last night contains a number of red herrings.  Firstly, the fact that John Bercow and his deputies don't vote has nothing to do with the government's real majority being 16 rather than 12.  Two Deputy Speakers come from the Labour ranks, which completely balances out Bercow and the other Deputy Speaker who are both drawn from the Tory benches.  It's purely and simply the failure of Sinn Fein to take up their seats that increases the de facto Tory majority by four.  Secondly, absences caused by pairing arrangements can't possibly increase the government's majority - the whole purpose of pairing is to ensure that there is no impact on the majority in either direction.

On the tax credits division, Labour, the SNP, the Lib Dems, the DUP, Plaid Cymru, the SDLP, the UUP and the Greens all voted against the government.  Douglas Carswell of UKIP voted with the Tories, as did - somewhat bizarrely - the independent Northern Ireland unionist Lady Hermon, who normally votes with Labour much more regularly than her unionist colleagues do.  That means the 'natural' government majority on the vote ought to have been roughly 20.  In reality, it should have been lower still, because there was a minor Tory rebellion.  The fact that the actual majority was as high as 35 can only be explained by non-paired absences on the opposition benches.

For all the DUP's dark hints about the consequences of Labour appointing a Shadow Chancellor who called the IRA "heroes", they don't seem to have had any qualms about voting in the opposition lobby last night.  I suspect nothing much is really going to change - it's far, far easier for Labour, the SNP and the DUP to find common cause in opposing specific Tory policies (often for completely different reasons) than it would have been for them to agree on a joint programme for government.

*  *  *

At PMQs today, David Cameron challenged Angus Robertson to produce a list of ways in which the "Vow" has been broken.  He conveniently waited until Robertson had used up both his questions before making that demand, so I hope that the SNP get the list ready in time for the next session.  It won't take long - it's been extremely well-documented that the Scotland Bill falls woefully short of the Smith Agreement in several key respects.

*  *  *

Dan Hodges is moving rapidly from unhinged to fascistic -

"If Jeremy Corbyn doesn't like the national anthem, fine. He wins an election, and changes it. Until then, he sings it."

Since when has the singing of the national anthem been a legal requirement? What the hell has happened to this "open", "diverse", "tolerant" country? It appears that it is now compulsory for anti-monarchist atheists to beg God to preserve the monarchy via the medium of song.

Corbyn's top appointments may all have been male, but they all made logical sense

I feel a bit sorry for Jeremy Corbyn after the flak he's taken over failing to appoint women to shadow any of the 'Great Offices of State'.  I'm sure he's serious about achieving gender balance at the top of politics, but if you look at the appointments individually, they all make perfect sense.

John McDonnell was the right choice as Shadow Chancellor, because the most common faultline in divided party leaderships is between the leader and the Chancellor/Shadow Chancellor.  In Chris Mullin's A Very British Coup, the nemesis of the left-wing Labour PM Harry Perkins is the moderate he appoints as Chancellor to reassure the City.  OK, that's fiction, but it's not hard to think of real world examples - Thatcher/Howe, Thatcher/Lawson, Blair/Brown.  It's probably better for the leader to avoid that problem by appointing a friend and ideological fellow traveller, and there is simply no female MP who fits that bill for Jeremy Corbyn - with the sole exception of Diane Abbott, who even her most fervent admirers would probably concede is unsuited to the role.

Just about the only thing Harriet Harman did as interim leader that could be described as a stroke of genius was to appoint Hilary Benn as Shadow Foreign Secretary.  He's by far Labour's best Foreign Secretary or Shadow Foreign Secretary since Robin Cook (although admittedly the competition isn't exactly stiff).  He's also a unifying figure, liked and respected across different wings of the party - a moderate, but one who proudly voted for his dad Tony Benn in the leadership and deputy leadership contests of the 1980s.  If he was willing to stay on in the role, surely Corbyn had no choice but to bite his hand off?

That only leaves Shadow Home Secretary, which by a process of elimination had to go to a defeated leadership candidate who was willing to serve.  It's scarcely Corbyn's fault that Yvette Cooper and Liz Kendall were both refuseniks, making the appointment of Andy Burnham inevitable.

There's been a lot of talk of 'tokenism' in relation to the female appointments Corbyn has made to the lower ranks of the Shadow Cabinet, but surely the true tokenism would have been to appoint the wrong person to one of the top jobs simply because she is a woman.  If all-female appointments had been made on merit to the top three posts, isn't that something we would celebrate, rather than fret about?

Stirring Survation survey suggests virtual dead heat on independence

We now almost have a full house of pollsters producing pre-anniversary polls on independence - of the six firms which traditionally poll on the subject now and again, only ICM haven't put in an appearance so far this month.

Should Scotland be an independent country? (Survation)

Yes 49.3% (+1.1)
No 50.7% (-1.1)

Although it's obviously encouraging that two of the three online firms that have reported over the last few days have shown a modest increase in the Yes vote, these numbers remain firmly within the familiar post-referendum range - Yes have previously been as high as 51% in a Survation poll.  So there's still no evidence of a surge for Yes over and above the one that took place in the immediate aftermath of last September's events.  The good news is that the initial surge was absolutely real and hasn't been reversed.  We know that because the Yes vote is significantly higher than 44.7% in this poll, in spite of the fact that Survation weight their results by recalled referendum vote.  Respondents who recall voting Yes have been weighted down from 413 to 385, and those who recall voting No have been sharply weighted up from 423 to 458.  Without that adjustment, Yes would be in the outright lead.

Even better are the clear signs that the progress for Yes since the referendum has come about - at least in part - because of direct switching by people who voted No.  John Curtice made a great deal the other day of the Panelbase poll showing that exactly the same proportion of Yes and No voters from last year would stick with their original choice, and that Yes had only made modest progress because it was in the lead among people who didn't vote in the referendum.  But that was actually an extremely unusual finding, and is contradicted by both YouGov and Survation.  In the Survation poll, 6.8% of No voters from 2014 would now vote Yes, compared with just 2.6% of Yes voters who would now vote No.

There's great news in this poll for those urging Nicola Sturgeon not to be scared of making a conditional commitment for a second independence referendum in the SNP's Holyrood manifesto.  A mere 4.1% of people currently planning to vote SNP say they would be less likely to vote for the party if another referendum is promised.  Even if you make the improbable assumption that every single one of those people would peel away, that would still leave the SNP with 51% of the vote on the constituency ballot.  (And in reality any minor losses could well be offset - 8.5% of prospective Labour voters say a referendum promise would make them more likely to vote SNP.)

Unfortunately, the poll was conducted just before Jeremy Corbyn was elected as Labour leader, so it can only offer hypothetical data on the potential Corbyn effect.  Among people who voted SNP this year, 14.3% say Corbyn would make them more likely to vote Labour, compared with 18.3% who say that they'd be less likely to do so.  That may sound encouraging from a nationalist point of view, but of course many of the people who claim they'd be less likely to vote Labour are probably hardcore SNP voters anyway.  What is more significant is that 16.9% of people who actually voted Labour in May say that Corbyn might put them off from sticking with the party next year.  In absolute terms, there are more SNP-voting respondents (58) who might be won over by Corbyn than there are Labour voters (33) who might be repelled, but that difference isn't great enough to make much of a dent in the SNP lead.  Having said that, we don't know exactly where the potential Labour defectors might go - in theory they might all go to other unionist parties.

A quick word about the YouGov poll, for which the datasets have now been released.  In spite of the fact that it was commissioned by a nakedly partisan client and thus contains leading questions, there is one absolutely devastating finding for the anti-independence camp.  Just 9% of respondents think that the notorious "Vow" which helped to win the referendum for No has been delivered in full.  A combined total of 52% say that only very little has been delivered, or nothing at all.

*  *  *

ALL-TIME HIGH FOR YES IN SCOT GOES POP POLL OF POLLS

There have now been enough recent polls to make it worthwhile updating this blog's Poll of Polls on independence for the first time in a full year, using the same method as before (although for obvious reasons only firms that have reported since the referendum are included - that means Opinium drops out of the sample).  Of the six polls taken into account, only the one from ICM is somewhat out of date, but given the strong indications that public opinion has been stable over the last few months, that may not make much difference.

Percentage changes are from the last update on September 17th last year, and almost certainly underestimate the progress made by Yes since then.  That's because four of the six firms have since introduced weighting by recalled referendum vote, which generally hurts Yes.

MEAN AVERAGE (excluding Don't Knows) :

Yes 49.6% (+1.5)
No 50.4% (-1.5)

MEAN AVERAGE (not excluding Don't Knows) :

Yes 46.0% (+2.4)
No 46.7% (-0.4)

MEDIAN AVERAGE (excluding Don't Knows) :

Yes 48.7% (+0.9)
No 51.3% (-0.9)

(The Poll of Polls is based on a rolling average of the most recent poll from each of the firms that have polled on independence since the referendum, and that adhere to British Polling Council rules. At present, there are six - YouGov, TNS, Survation, Panelbase, Ipsos-Mori and ICM. Whenever a new poll is published, it replaces the last poll from the same company in the sample.)

Why does the new Labour leader think Scotland is a lesser nation than Ireland?

A second quick note of the day, this time to let you know I have a new article at the International Business Times, about Jeremy Corbyn's double-standard on the national questions in Scotland and Ireland.  You can read it HERE.

Caledonian Consequences of Corbyn

Just a quick note to let you know that I have a new article at The National, about the potential impact of Jeremy Corbyn's victory on the electoral fortunes of both Scottish Labour and the SNP.  You can read it HERE.

Yonder YouGov poll sees SNP lead growing even further

Here's a paradox - we were much happier with the independence figures in the TNS poll than in the YouGov poll, but the new Holyrood voting intention figures from YouGov show further movement in favour of the SNP, whereas TNS suggested a recovery for Labour.

Constituency ballot :

SNP 51% (+2)
Labour 22% (-3)
Conservatives 18% (+3)
Liberal Democrats 4% (-3)

Regional list ballot :

SNP 45% (+2)
Labour 20% (-4)
Conservatives 18% (+2)
Greens 9% (+2)

Unlike the divergence that's opened up between the pollsters in relation to the independence question, this is a straight contradiction on the direction of travel.  It's not just the swing from Labour to SNP that is inconsistent with TNS - the mini-surge for the Tories, taking them to the improbable position of being just 2% behind Labour on the list, is the opposite of what TNS showed.  The most straightforward explanation is that nothing much has changed and that we're simply seeing the freakish effects of sampling variation.

More importantly, although YouGov's fieldwork is more recent than that of TNS, it still predates Jeremy Corbyn's election as Labour leader.  So this poll doesn't take us any further forward on the crucial question of whether or not this weekend's development has been a game-changer.  But at least we can be confident that the SNP arrived at this landmark moment in the best possible shape imaginable.

Panoramic Panelbase poll produces steady picture on independence

Panelbase have become the fourth pollster to conduct an independence poll to mark the run-up to the anniversary of the referendum - will ICM and Survation join the party over the coming days?  Today's poll removes any lingering doubt as to the explanation for TNS and Ipsos-Mori both showing a clear majority in favour of independence - it appears that was caused by different methodology, rather than by a change in public opinion over the summer.

Should Scotland be an independent country? (Panelbase)

Yes 47% (n/c)
No 53% (n/c)

I'm slightly troubled by John Curtice's reaction to this poll and yesterday's YouGov poll.  He quite fairly makes the point that they undermine the idea of a very recent swing to Yes, but you'd think an equally important point to make is that we're now left with no idea whether Yes or No are ahead right now, or indeed whether Yes or No have been ahead for much of the last year.  The only conclusion that is consistent with all of the polls we've seen is that the state of play has been relatively steady following a Yes surge (of uncertain scale) in the immediate aftermath of the referendum, but beyond that we have a clear split between Ipsos-Mori and TNS on the one hand, who are showing a Yes lead, and YouGov, Panelbase, Survation and ICM on the other hand, who are (mostly but not always) showing a slender No lead.

Instead, the subtext of what Curtice goes on to say seems to be that more credence should be given to the No lead shown by Panelbase and YouGov, because both firms weight by recalled referendum vote - something that TNS and Ipsos-Mori don't do.  This overlooks (or at least downplays the significance of) three vitally important points -

1) The failure to weight by recalled referendum vote is not the only factor that distinguishes TNS and Ipsos-Mori from the other firms, or even the most important one.  They are also the only pollsters that use a 'real world' data collection method, meaning they actually go out and find a fresh sample of respondents for every poll - in contrast to the other firms who are reliant on volunteer online polling panels.  That difference in itself could easily explain the divergence we are seeing - it could be that there is now a real world majority for independence that the online firms are simply unable to pick up.

2) TNS do weight by past vote recall, but they use the 2015 general election as their reference point, rather than the referendum.  It's not at all clear that switching to referendum weighting would be any better at correcting for the presence of slightly too many 'eager Nats' in the sample.

3) The Yes lead in the Ipsos-Mori poll was so big that it stretches credibility to suggest that weighting by recalled referendum vote would have magically produced an outright No lead in line with the online firms.  Not impossible, admittedly, but unlikely.

*  *  *

I was planning to update this blog's independence Poll of Polls last night for the first time since the referendum, because we now have more than enough recent polls to make it a meaningful exercise.  But I suddenly realised that there's a discrepancy in the reported figures from the YouGov poll.  The What Scotland Thinks website says that with Don't Knows included, the numbers are Yes 43%, No 49% - which, even taking account of rounding issues, is completely irreconcilable with the headline figures of Yes 48%, No 52% with Don't Knows excluded.  Hopefully the mystery will be solved when the datasets emerge.