The unionist media are in denial about it, but a second independence referendum moved even closer this week

As far as I can remember this has never happened before (which tells its own story), but it turns out that I owe a mainstream media journalist a comprehensive apology for taking him to task on Twitter a few months ago for what I took to be spin on behalf of the unionist establishment.  Not long after the EU referendum, Mark Mardell of the BBC claimed that the SNP were set to join the "forces of soft Brexit".  I retorted that this was a plainly ridiculous notion, because the SNP's stated aim was to keep Scotland within the European Union in line with how the country had just overwhelmingly voted.  But Mardell was completely right and I was completely wrong.  There's no two ways about it - Nicola Sturgeon has clearly modified her position somewhat, and the detailed plan she put forward the other day indicates a willingness to accept our enforced departure from the EU, albeit only if a great many extremely stringent conditions are met.

However, in spite of misjudging the trajectory a few months ago, I still feel extremely confident in saying this : the likes of Kenny "Devo or Death / Brexit or Bust" Farquharson and whoever it is that writes the Guardian editorials are deluding themselves when they imagine that there is a realistic chance we are heading for a compromise that will head off a second independence referendum.  Theresa May took only a few hours to reject Sturgeon's proposals out of hand, so to believe in the chance of a deal you must also believe that what we saw this week was merely some kind of way-station on the Scottish Government's journey towards accepting a semi-hard Brexit and departure from the single market, rather than a statement of Sturgeon's absolute bottom line.  The Greens (who have disorientated us all by reinventing themselves as the new indy fundamentalists) have demanded that it must be the latter, and I'm convinced that's how it will be.

I've never met Nicola Sturgeon (I've never even had a selfie taken with her, so I really am the lowest of the low), and therefore can't read her mind.  I've heard it said, as I'm sure we all have, that she's instinctively more cautious than Alex Salmond, although I also remember hearing it said in 2012 that she was actually much more bullish than Salmond about the all-or-nothing gambit of calling a single-question referendum on independence without a Devo Max option on the ballot paper.  Regardless of the truth about her temperament, though, the real reason why a second independence referendum now looks almost inevitable is that she has to take her party with her when she makes the final decision.  It's just a non-starter to imagine that the SNP rank-and-file are going to accept calling off the dogs on indy in return for departure from the single market, and no special status within Europe for Scotland - which is all that will be on offer from the UK government, barring a wildly improbable climbdown.  The only way I could imagine the membership signing up to that 'deal' would be if there is some kind of compensatory offer involving a massively beefed-up devolution settlement - but London seem to be showing zero interest in that idea as well.

The unionist media are, as is their custom, telling themselves precisely what they want to believe - that the SNP membership don't matter, and that Sturgeon will be guided solely by opinion poll numbers and a private acceptance of the economic importance of an entirely fictional concept known as "the UK single market".  It may be that it will continue to be possible to portray the opinion poll evidence as discouraging for Sturgeon over the coming months - Scottish polls are largely commissioned by anti-independence clients, after all, which means that the supplementary questions are often framed in a way that will produce the most negative answers.  But that will not be the determining factor on whether a referendum goes ahead.  Public opinion at the end of both the independence and Brexit referendums bore little resemblance to public opinion at the start of the campaigns.  It would be a brave unionist who felt sure that initial Yes support in the mid-to-high 40s will not prove to be a successful springboard to victory.  Which is a point well worth pondering - do the forces of unionism seem relaxed to you, in the way they would be if they really thought a referendum was unwinnable for Sturgeon?  No, they don't seem relaxed to me either.

*  *  *

Theresa May seems to have an unconscious desire for a career in comedy.  Only two years ago, her party told Scots to vote No because it was supposedly the only way to keep Scotland in the European Union.  As a direct result of the electorate falling for that line and voting No, we are now on course to be forced out of the EU against our will.  And yet Ms May innocently says that there is still "no need or reason" to revisit the decision on independence.  It's a bit like someone saying to you "OK, I will marry you, but ONLY if you never, ever steal ANY of my chocolates EVER again", and you absolutely promise to go along with that, and then two months later you express bafflement about her decision to call the whole thing off while you're busy munching on her After Eights.

Yes, "The Vow" could have been delivered - but the UK government chose not to do it

I've been meaning to make an additional point about the UK government's outrageous legal arguments relating to the Sewel Convention. I noticed on Twitter a few days ago that some independence supporters seemed to be getting two different concepts muddled up, and as a result were grotesquely (albeit unwittingly) giving the Tories something of a free pass. Basically the confusion relates to the words 'unenforceable' and 'repealable'. There is a strong argument (not unchallengeable, but strong) that it would be next-to-impossible to enshrine the Sewel Convention into law in a way that is unrepealable, simply because of the principle of absolute parliamentary sovereignty. Any law can in theory be repealed by simple majority vote. But the crucial point is that for as long as any such law is on the statute book, it ought to be perfectly enforceable.

To give an example : the Canada Act 1982 terminated Westminster's power to pass laws for Canada. Even if you argue that the principle of parliamentary sovereignty theoretically means that decision can be unilaterally reversed (a point that might be accepted by the UK courts but certainly not by the Canadian ones), there would actually have to be an "Act to Repeal the Canada Act" before Westminster could return to regulating the drains in Toronto, or whatever. So here's the problem - the Scotland Act 2016 didn't even get us that far in respect of Sewel. It used self-sabotaging language to render itself unenforceable, even while it's on the statute book. That was a deliberate choice of the UK government - there was nothing inevitable about it, and they have quite openly boasted in the Supreme Court about how the law was framed to ensure that nothing whatever changed (in particular by the insertion of the words "it is recognised" and "normally").

It's a distinction we must be absolutely clear about. That the Sewel Convention is not unrepealable is arguably part-and-parcel of the UK's most fundamental constitutional principle. But that the Sewel Convention is unenforceable right now was a political choice made by the London government - and that choice was made in brazen contravention of the "Vow" that secured the No vote in September 2014. They mustn't be allowed to get away with the fiction that an unenforceable declaration of commitment to Sewel was the best they could possibly have done within the confines of the constitution, because that simply isn't true.

*   *   *

Even by his own exalted standards, Mike "can't be arsed" Smithson of Stormfront Lite fame has been making some dubious comments of late, so I just thought I'd take a moment to deal with a couple of his daftest points.  Firstly, he ludicrously claimed last week that the polls on Brexit were not actually wrong, and that anyone who says otherwise is talking "rubbish".  To support this notion, he pointed out that 59% of polls in the last month of the campaign showed "non-Remain leads" (meaning either a tie or a Leave lead).

There's a hole in that logic that a poorly-educated hamster could see through.  If public opinion changes during an election campaign, as it clearly did in the run-up to polling day in June, it will very often be possible to artificially select a time-frame that makes the polls look much more accurate than they actually were.  For example : everyone knows that the polls (with a single honourable exception) got the 1970 general election hopelessly wrong.  They pointed to a Labour overall majority, and the real outcome was a Conservative overall majority.  But between 1967 and 1969, the Tories had actually been way in the lead in the polls, before Labour seemingly turned things around.  Mike Smithson would doubtless argue that, contrary to the universally-held perception, the 1970 polls were not wrong, because over a three-year period they showed the Tories ahead on average.  That would technically be accurate, and is no more or less silly than what he's said about the Brexit polls.

To credibly pray in aid a poll that gets the result broadly "right" a month before polling day, there must at least be an absence of evidence that public opinion changed significantly in the intervening weeks.  Unfortunately for Smithson, the evidence that there was a late swing to Remain as June progressed is overwhelming.  In early June, YouGov "correctly" gave Leave a 4% lead, but the eve-of-referendum poll from the same firm put Remain 2% ahead, and the on-the-day poll saw the Remain lead extend to 4%.  Other firms showed a similar trend.  The likelihood is that all of those polls were inaccurate, and that when Leave appeared to be slightly ahead, they were actually ahead by a significant margin.

It seems Smithson would like to think that the high level of advance postal voting is an adequate excuse for the inaccuracy of the later polls - but it isn't.  If people had already voted by post, the polling firms asked them how they had actually voted, rather than how they planned to vote.

*   *   *

Elsewhere, Smithson rather desperately used aggregate numbers from a tiny handful of by-election results to demand that the BBC should revert to treating his own party (the Lib Dems) as more important than UKIP.  Now, it may well be that the BBC have been guilty of giving UKIP disproportionate coverage over the years and thus unleashing the Brexit monster, but what's done is done - they can't now ignore the fact that UKIP received almost one-and-a-half million more votes than the Lib Dems in the general election, and have remained ahead of the Lib Dems in most opinion polls over recent months.  Even since Richmond Park, there has been a poll giving UKIP a slight lead over the Lib Dems.

I would hope that somewhere, deep down, Smithson has just enough self-awareness to be embarrassed about his wildly unrealistic demands on behalf of his own party - especially bearing in mind his offensive and downright Orwellian claim a few years back that the SNP were guilty of "censorship" for simply trying to gain fair access to the televised leaders' debates.

Lashings of loveliness in Leven as SNP score key by-election triumph

Leven, Kennoway and Largo by-election result, 15th December 2016 :

SNP 37.0% (-4.0)
Labour 28.4% (-6.9)
Conservatives 18.5% (+11.7)
Liberal Democrats 14.3% (+4.3)
Greens 1.8% (n/a)

Things have come to a pretty pass when Labour are quite genuinely 'celebrating' a defeat in a former heartland, a 7% drop in their own vote share, and a 1.5% net swing from Labour to SNP.  But it's probably fair to say that, in the context of the times, this is a tolerably good result for them.  If extrapolated nationally on a 'just a bit of fun' basis, it would put them only a few points behind the SNP.  The snag is, of course, that there is absolutely no evidence from any opinion poll or from any other recent by-election to suggest that their resilience tonight is anything more than a freakish one-off.  The chances are that there were local or personality factors that worked in their favour in this particular contest.

Should we be at all concerned about the slippage in the SNP vote, given that the party had only a modest national lead in the last local elections in 2012?  A loss of support is certainly more noteworthy in this sort of by-election than it would be in areas of traditional Tory strength - we can't just neatly put it down to a small number of No-voting ex-SNP supporters finding a way of resolving their cognitive dissonance.  However, the most recent national opinion poll (conducted only two or three weeks ago) gave the SNP almost half of the popular vote, and in spite of the question marks that now hang over the polling industry, it's unlikely that polls would completely fail to detect a dramatic shift in public opinion.

The small increase in the Liberal Democrat vote is very much in line with last week's Scottish local by-election, the parliamentary by-election in Sleaford and North Hykeham, and also a couple of Britain-wide opinion polls.  So there does seem to have been a genuine post-Richmond Park bounce for the party.  The problem is that they'll need to follow up that win at some point if they want to maintain their momentum, and respectable third or fourth place finishes aren't really going to be enough.  The afterglow of a single by-election success - no matter how spectacular - is generally pretty short-lived, and if what we're seeing now is as good as it gets for the Lib Dems, they're not really back in the game in any meaningful sense.

Heartbreak for Rennie as Carnoustie by-election fails to show much trace of a Richmond Park bandwagon effect

Immediately after the Richmond Park result became clear a few days ago, Professor John Curtice speculated about the possibility of an 'Eastboune effect' - a reference to a by-election gain out of the blue in 1990 that got the Lib Dems back into the game after a couple of years during which they had appeared to be fading into irrelevance.  Well, the first little test of whether we're about to see something similar came in yesterday's Carnoustie by-election, and although there was an increase in the Lib Dem vote, it was pretty trivial in scale.

Carnoustie & District by-election result, 5th December 2016 :

Independent - Cheape 43.5% (n/a)
SNP 32.1% (-4.2)
Conservatives 17.7% (+11.6)
Labour 4.4% (-1.8)
Liberal Democrats 2.3% (+1.4)

I think the reason why a full-blown Eastbourne effect is unlikely is that British politics is much more fractured than it was in 1990.  Not only are there more 'major' parties to choose between, but Scotland is also now much more psychologically detached from the rest of the UK.  People here will have regarded the Richmond Park result as almost irrelevant to us - if you want to back an unashamedly 'Remain' party here, you'll generally do it by voting for the SNP, and not for a party that incomprehensibly voted against European single market membership in the Scottish Parliament only the other week.

That said, there's clearly scope for a 'niche' Lib Dem comeback in parts of England with decent levels of Remain support.  I was shaking my head in disbelief on Thursday night at Diane Abbott's complacent line of argument that the Lib Dems' pro-European pitch will not play well in areas that voted to Leave.  It doesn't need to, Diane - they're not trying to win a general election.  46% of the English electorate voted Remain, and if the Lib Dems can win over just one-third of those people, they'll essentially double their vote and regain a number of their old seats.  Polls prior to Richmond Park suggested that the "Brexit Means Brexit" Tories were still comfortably outpolling the Lib Dems among Remain voters in England.  There is now a golden opportunity for the Lib Dems to reverse that.

To return to Carnoustie, the SNP vote has actually held up much better than in the by-election in the same council area last week.  There seems to be a pattern in the north-east of some No voters switching from the SNP to the Tories, but it hasn't happened on a big scale on this occasion - most of the boost in the Tory support came from elsewhere.  Technically, this result is an independent gain from the SNP, but as is so often the case in STV by-elections, that's pretty meaningless - the SNP didn't actually win the popular vote in the ward last time around (they finished second behind a different independent).

As was the case last week, when the lower preferences of Labour voters were distributed, there were more transfers to the SNP (35) than to the Tories (20).  Has Kezia backed the wrong horse?

UK government boasts that "The Vow" was a gigantic con-trick

As we all remember, a key part of the panic-stricken last-minute "Vow" that helped to secure a narrow No victory in the independence referendum was the promise that the devolved Scottish Parliament would be made "permanent".  Implicit in that pledge was that the Sewel Convention (which, among other things, prevents the powers of the parliament from being altered without the consent of MSPs) would be put on a statutory footing.  Without that safeguard, "permanence" would plainly be meaningless - the parliament could be left as an empty shell, unilaterally stripped by Westminster of all or most of its powers.

It was widely noted that, technically, it was next to impossible to guarantee either the permanence of the parliament or the inviolability of the Sewel Convention, because the sovereignty of the Westminster parliament means that any constitutional law can simply be repealed later on.  Nevertheless, the UK government insisted that the Scotland Act 2016 provided as much assurance as was humanly possible to give within the UK constitutional framework.

A number of us were a tad sceptical about that, and couldn't help wondering whether the use of weird and seemingly redundant wording within the legislation such as "it is recognised that" and "the UK Parliament will not normally legislate without consent" were deliberately intended to be self-sabotaging, and to render the whole thing unenforceable.  Not at all, we were told. That was just paranoia.  Yet more SNP grudge and grievance.

Hmmm.  As it turns out, all it's taken is eight months since the Scotland Act passed into law, and the UK government are already openly admitting that we were correct in each and every respect about the cynical game they were playing.  So desperate are they now to head off the risk of the Supreme Court granting the Scottish Parliament the kind of say over the Brexit negotiations that might actually befit "the most powerful devolved parliament in the world" (ahem), they've dropped all the former pretence, and have submitted a legal argument that explicitly argues that the wording of the relevant section of the Scotland Act deprives it of all credibility.

"The legal irrelevance of the Sewel convention is expressly accepted"

"the convention does not purport to prescribe an absolute rule. Its content is only that “Westminster would not normally legislate” (emphasis added). Whether circumstances are ‘normal’ is a quintessential matter of political judgment for the Westminster Parliament and not the courts. There are no judicial standards by which to measure such a question..."

"Nothing in that analysis is affected by the amendment of s. 28 of the Scotland Act 1998 (by s. 2 of the Scotland Act 2016) to include: “(8) But it is recognised that the Parliament of the United Kingdom will not normally legislate with regard to devolved matters without the consent of the Scottish Parliament”. All s. 28(8) does is to recognise the terms of the political convention in legislation. That does not render the application of it in any particular instance a justiciable matter for the courts. It is trite that legislation may include provisions which do not give rise to justiciable legal issues. The content of s. 28(8) is the same as that of the convention, save that its purely political nature is further emphasised by (a) the opening wording that it is “recognised”, and (b) its placement immediately after s. 28(7) which affirms the unconstrained legislative competence of the Westminster Parliament."

In plain language, this is a boast that the supposed placing of the Sewel Convention on a statutory footing was a con-trick. Further, it's an invitation to the Supreme Court to confirm that the deception was pulled off successfully. That whole section of the Scotland Act, we're being told, was the equivalent of a pretty illustration in a textbook - ie. for decorative effect only. At best, it was like forgoing a marriage certificate in favour of a small tattoo saying "Jenny and Kevin 4eva".

The eccentric notion that the government which crafted the law, and not the courts, should get to decide how to interpret the meaning of the word "normally" reminds me a touch of the Führerprinzip in Nazi Germany (ie. the government's word is above the written law), or the right of the communist Chinese National People's Congress to interpret the Hong Kong Basic Law. The rule of law in democratic countries does generally rest on a basic separation of powers - the political legislature passes the law, and then the non-political courts interpret and enforce it. That is the only way of ensuring non-arbitrary application of the law.  Apparently, Westminster's exercise of overlordship in Scotland is exempt from that general principle.

The UK government's lawyers could have saved themselves a lot of time by submitting a legal argument that simply read : "OUR CONTEMPT FOR THE PEOPLE OF SCOTLAND IS ABSOLUTE. WE LIED, WE CHEATED, AND WE NO LONGER CARE WHO KNOWS ABOUT IT. SUCK IT UP, JOCKS."

You've dunnit again, Gov.

Just a quick note to let you know that I have a new article at the TalkRadio website about yesterday's YouGov poll on independence.  It also rehearses some of the many shortcomings of YouGov's polling during the indyref.  You can read the article HERE.

"Engrossing" YouGov poll finds that Scotland is behind Sturgeon's drive to remain in the EU after the UK leaves - and understands this will require independence

A new full-scale Scottish YouGov poll is out today, and the most significant finding is that there is narrow majority support (albeit within the margin of error) for Scotland seeking to remain in the European Union after the UK leaves...

Would you support or oppose Scotland seeking to negotiate with the European Union to remain part of the EU after the rest of Britain leaves?

Support 42%
Oppose 41%

The subsequent question reveals that, by a massive 62%-22% margin, people understand that independence will be required to remain in the EU after Brexit.

There's particularly vexing news for Kezia Dugdale here.  As you'd expect, SNP voters are firmly in favour of retaining Scotland's EU membership, and Tory voters overwhelmingly follow the 'Brexit Means Brexit for everyone' doctrine.  But Labour voters are essentially split down the middle on this point.  It appears that Dugdale may have miscalculated in her belief that the rump Labour support (which is all she appears to care about holding onto) shares the hardline unionist DNA of the Tory support.  Incredible though it may seem, therefore, there are clear grounds for optimism that even more Labour voters could be won over to the SNP if Dugdale maintains her inflexible stance on leaving the EU.

The poll was commissioned by a rabidly anti-independence client (The Times), who are naturally keen to draw attention to the portions of the poll they find more palatable.  In particular, they seem to be beside themselves with excitement to discover that, for the first time since 2014, a YouGov poll has found that headline support for independence is fractionally lower than it was in the indyref.  (The operative word here is 'fractionally' - the Yes vote in the referendum was 44.7%, and YouGov currently has it at 44%.)  But the reality is that the general pattern of recent times has been a Yes vote that hovers only a little higher than the 2014 result, with a figure of 47% being particularly typical.  If that's roughly where we are, you'd fully expect the odd individual poll to put Yes slightly below 45%, simply due to sampling variation (ie. margin of error).  It's theoretically possible that this poll is the first to detect a genuine dip in support, but there's absolutely no reason to jump to that conclusion - unless of course you're indulging in wishful thinking.

By the same token, The Times are ascribing significance that simply cannot be statistically justified to minor changes on the supplementary questions.  For example, opposition to a second independence referendum being held before the UK leaves the EU (ie. within a very tight two-year timescale) has increased since the poll in August from 50%-37% to 54%-35%.  But that change can be easily explained by the margin of error - if, say, the true 'oppose' figure has remained steady at around 52%, the margin of error means that the reported figure in individual polls would be expected to be anywhere between 49% and 55%.  So The Times' liberal use of words like 'slump' is, I'm afraid, a tad over-excitable.

One thing to watch out for today : YouGov have, frankly, become notorious over the years for their lack of objectivity on the topic of Scottish independence.  After their last poll in August, they posted a ludicrous article which brazenly misrepresented their own results.  They stated baldly that a majority of Scots were opposed to a second referendum, but, in fact, respondents hadn't even been asked such a broad question - they'd only been asked whether they wanted a referendum before Britain leaves the EU, which could be very soon indeed.  YouGov also stated that, if and when a referendum takes place, a majority would vote No again.  There was absolutely nothing in their results that would even begin to justify such a wild claim.  The headline question on independence had merely asked, in line with normal practice, how people would vote on independence if a referendum was held tomorrow.  Both of those facts also apply to today's poll, so just keep an eye out in case YouGov attempt the same misrepresentation on this occasion.  At time of writing, they haven't so far.

One small piece of credit I can give YouGov, however, is that they've finally put their house in order and started interviewing the correct electorate - ie. over-16s, rather than just over-18s.  So we no longer have to worry about Yes and SNP support being underestimated for that specific reason.  It's very hard to understand why it's taken such a ridiculously long time for basic good practice to be followed - but better late than never.

It shouldn't be overlooked that today's poll makes fairly grim reading for Theresa May personally.  Her net personal rating has collapsed from +13 in August to -5 now.  Margin of error 'noise' certainly can't explain such a big change.  The Prime Minister still has a long way to go before she reaches Thatcher-style depths of unpopularity, but on her current trajectory (and bearing in mind that her honeymoon period has only really just ended), it's perfectly possible she could eventually arrive at that destination.  If she does, the consequences for opponents of independence could be catastrophic.

The final question of the poll is a bit of an oddity.  Unionists have gleefully leaped on it as proof that 'only' 13% of the population have been involved in the SNP's recent consultation process (which would actually be a pretty impressive figure).  But the wording of YouGov's question leaves a lot to be desired.  The survey is cited as the "National Conversation", whereas to the best of my knowledge it's actually been generally referred to as the "National Survey".  Respondents are also asked whether they have been "approached to take part" in the survey, which on the face of it would exclude anyone who took part without being "approached".  Whether the wording of the question is just clumsy, or whether it's deliberately intended to muddy the waters, is hard to say.

A general point that needs to be made about all polling at the moment : regardless of whether it's good or bad, we simply can't be sure it's reliable.  Fergus Ewing was asked about the headline independence results today, and he pointed out that polls in recent elections and referendums had mostly been wrong.  In years gone by, a politician trying to rubbish the polls would have been regarded as a bit desperate, but as things stand it's hard to deny he's got an excellent point.  It's particularly worth taking a look at the huge leads the Remain campaign had in telephone polls for such a long time.  No-one can say with any great confidence what the true support for independence is right now - let alone what it will be in a few months' time, or in a couple of years.  That's a statement of fact.  The era during which it was rational for political leaders to make strategic decisions on the basis of a couple of percentage points here or there in opinion polls is well and truly over.  At best, the polls of today are a ball-park guide - and that park seems to be getting ever bigger.

Amazement and astonishment in Arbroath, Angus as buoyant SNP bag a by-election belter

Not entirely sure why a by-election was held on a Monday - it's deeply unnatural, like an accurate John McTernan prediction, or something like that.  Anyway, here is the result...

Arbroath East and Lunan by-election result, 28th November 2016 :

SNP 35.0% (-8.8)
Conservatives 27.0% (+12.2)
Independent - Speed 17.2% (n/a)
Independent - Smith 11.8% (n/a)
Labour 6.7% (-6.0)
Liberal Democrats 2.3% (-0.6)

At first glance, that looks like a routine SNP hold on a reduced majority - but as is so often the case with STV by-elections, it's not that simple.  The vacancy was caused by the resignation of an independent councillor - so it's technically an SNP gain from independent, even though the independent was 18% behind the SNP in the popular vote in the ward last time around.  According to the 'Mike Smithson Doctrine', which ludicrously takes no account whatever of the previous result in the ward, this is therefore a result of unalloyed wonderfulness for the SNP (which is probably why Smithson hasn't mentioned it).

Back in the real world, there has been a significant swing from SNP to Tory, which very much follows the pattern of realignment we've been seeing recently - with Yes-voting ex-Labour supporters in west-central Scotland moving en masse to the SNP, and to a lesser extent No-voting ex-SNP supporters in the north-east moving to the Tories.  In a proportional representation election, that swap will always work out quite well for the SNP, but it could potentially lose them a few seats in the next first-past-the-post Westminster contest.  In retrospect, it seems a minor miracle that the post-indyref swing to the Tories in their former heartlands was delayed long enough for the SNP to clean up quite so comprehensively in last year's general election.

As the SNP candidate was elected on the sixth count, we're able to see where the various transfers went.  It's refreshing to discover that Scottish Labour's love affair with the Tories isn't shared by a majority of their own voters in Arbroath - 31 Labour transfers went to the SNP, and only 18 to the Tories.

Scot Goes Pop mourns the passing of Fidel Castro

We interrupt your normal programming with a special message for our resident troll, who had this to say earlier -

"Not even a mention of Fidel on this fash right wing Nat si blog."

Consider Fidel mentioned by this commie left-wing anti-YoonYoon-sis blog.

Actually, the serious answer is that the record of Castro and Cuban communism is a mixed one.  There have been appalling human rights violations, and Cuba is now basically the only outright dictatorship in the American hemisphere - but there have also been extraordinary gains in health care and literacy.  Which side of the coin matters more?  They both matter.

To state the bleedin' obvious : yes, the US presidential polls were wrong

As GA Ponsonby wittily put it a couple of hours ago -

"The social media relationship between James Kelly and Stuart Campbell has broken down. Yet another reason for Humza to resign quite frankly."

Yes, it's true - after God knows how many years of following each other on Twitter and having a very amicable relationship, Stuart Campbell of Wings Over Scotland abruptly blocked me today.  He had become increasingly angry after I challenged his claims that the opinion polls in the US presidential election were not especially inaccurate, and he eventually told me in trademark fashion to "f*** off".  I have to say I find this a very sad development - over the years, I've backed him to the hilt over the totally unfounded allegations of misogyny and other assorted forms of bigotry, and to be fair he's also stood up for me on a number of occasions.  But throughout my near-decade of writing this blog, I've always felt very strongly that it's important never to let the 'patronage' of a leading blogger deter me in any way from pointing out when I think that person has got something wrong.  I did it with Mike Small in January.  I've actually done it a few times with Stuart before (for example in a debate about the morality of the Hiroshima bombing), and he has always previously reacted in a very constructive and mature way.  For some reason I simply don't understand, today was different.

I wasn't planning to make any further comment on the exchange, but the nature of having a dispute with someone who has several times as many followers as you do is that some of those people pile in after the event, demanding that you answer certain points.  So, free of the 140-character restraint on Twitter, here is my response.

One thing I've felt about Stuart for quite some time is that he fundamentally misunderstands the nature of the standard 3% margin of error in public opinion polls.  I seem to recall that after last year's polling disaster at the UK general election, he argued that the polls hadn't really failed, because they averaged out at a level-pegging race, which was more or less within the margin of error of the commanding 7% Tory lead we ended up with.  But that simply isn't how it's supposed to work.  The margin of error takes account of one type of error, and one type only - namely error caused by random sampling variation.  It assumes that everything else - demographic targets, weighting, etc - will be absolutely bang-on correct.  What that means in practical terms is that, although small errors in individual polls will always be commonplace, they should be distributed in a fairly random way.  If, for example, the polls had been correct to within the margin of error at the general election (leaving aside the complicating factor of a possible late swing), we might have seen a pattern in the final polls along the following lines...

Poll 1 : Tory lead of 5%
Poll 2 : Tory lead of 11%
Poll 3 : Tory lead of 5%
Poll 4 : Tory lead of 7%
Poll 5 : Tory lead of 6%
Poll 6 : Tory lead of 8%
Poll 7 : Tory lead of 3%
Poll 8 : Tory lead of 8%
Poll 9 : Tory lead of 6%
Poll 10 : Tory lead of 5%

In that hypothetical example, six out of ten polls underestimate the Tory lead, three overestimate it, and just one is absolutely correct.  That's the sort of thing that can easily happen by random chance.  But what you're NOT seeing there is every single poll underestimating the Tories, and almost all of them doing it by an amount that is either at the extreme end of the margin of error, or that exceeds the margin of error.  According to Wikipedia, and excluding a Survation poll that conveniently only appeared after the election result was already known, these were the actual last ten polls of the 2015 campaign -

Populus : Tied race
SurveyMonkey : Tory lead of 6%
Ashcroft : Tied race
Ipsos-Mori : Tory lead of 1%
YouGov : Tied race
ComRes : Tory lead of 1%
Survation : Tied race
ICM : Labour lead of 1%
ICM : Tied race
Panelbase : Labour lead of 2%

Not only did all of those polls underestimate the Tory lead, but the majority of them did so by slightly more than the margin of error.  That is not the sort of pattern that is remotely likely to occur by random chance - which tells us that the error wasn't primarily caused by the sampling variation allowed for by the margin of error, and that significant methodological mistakes were probably to blame.  (Again, that conclusion leaves aside the possibility of late swing, but it's probably correct to do so, given that YouGov's on-the-day poll was wildly inaccurate.) 

What Stuart would say, and what he effectively did say eighteen months ago, is that because on average the final polls 'only' underestimated the Tory lead by around six or seven points, they were basically accurate to within the margin of error (ie. with Labour overestimated by around 3% and the Tories underestimated by around 3%).  That just doesn't stack up.  The 3% margin of error only applies to each individual poll.  Random statistical noise should to a decent extent balance itself out over a large batch of polls, leaving you with a much smaller error.  In my hypothetical example above, the polling average underestimates the Tory lead by only 1% after rounding, which is much more typical of what you'd expect if the polls were essentially 'right'.

In my exchange with Stuart today, I was only really interested in disputing his points about margin of error, but he tried to sidetrack me into discussing other factors - in particular that polls are snapshots not predictions, and that the US presidential election is not decided by the national popular vote.  Technically, those are reasonable points to make, but when you put them together to try to construct a case that the polls didn't really get it wrong, you do start to get into the realms of the fantastical.  According to the final polls, Hillary Clinton had a national lead of around 4% going into election day.  It is simply not credible to claim that Trump could have won the election if that had actually been the result.  In any case, the possibility of a freak outcome in the electoral college is precisely what the state polls are there to warn us about - and they were even more inaccurate than the national polls!

As far as very late swing is concerned, yes, that can happen, but it won't generally be on an enormous scale, and it should show up in the exit polls (the only polls that are genuinely predictions, rather than shapshots of opinion).  As you can see HERE, the exit polls pointed to a clear Clinton victory in the electoral college.  In the vast majority of states polled, Trump was underestimated.  In almost half of the states, he was underestimated by a greater amount than the margin of error could - even theoretically - explain.

No matter how big the error, it's always possible to attempt to cobble together some kind of tenuous narrative that gives the polling firms a free pass.  If a 40% Labour lead vanishes into thin air on polling day, you can argue that 20% of voters may have changed their minds at the very last second.  But in the real world, there comes a point where you have to accept that the emperor has no clothes, and that the polls were just plain wrong.  They were wrong on Netanyahu, they were wrong on Cameron, they were wrong on Brexit, and they were wrong on Trump.  As I acknowledged the other week, everything is relative, and I would still pay much more heed to polls than to other so-called 'predictors' such as betting and financial markets.  But as of this moment, polls are plainly nowhere near as reliable as they are supposed to be.