Tuesday, November 29, 2016

Amazement and astonishment in Arbroath, Angus as buoyant SNP bag a by-election belter

Not entirely sure why a by-election was held on a Monday - it's deeply unnatural, like an accurate John McTernan prediction, or something like that.  Anyway, here is the result...

Arbroath East and Lunan by-election result, 28th November 2016 :

SNP 35.0% (-8.8)
Conservatives 27.0% (+12.2)
Independent - Speed 17.2% (n/a)
Independent - Smith 11.8% (n/a)
Labour 6.7% (-6.0)
Liberal Democrats 2.3% (-0.6)

At first glance, that looks like a routine SNP hold on a reduced majority - but as is so often the case with STV by-elections, it's not that simple.  The vacancy was caused by the resignation of an independent councillor - so it's technically an SNP gain from independent, even though the independent was 18% behind the SNP in the popular vote in the ward last time around.  According to the 'Mike Smithson Doctrine', which ludicrously takes no account whatever of the previous result in the ward, this is therefore a result of unalloyed wonderfulness for the SNP (which is probably why Smithson hasn't mentioned it).

Back in the real world, there has been a significant swing from SNP to Tory, which very much follows the pattern of realignment we've been seeing recently - with Yes-voting ex-Labour supporters in west-central Scotland moving en masse to the SNP, and to a lesser extent No-voting ex-SNP supporters in the north-east moving to the Tories.  In a proportional representation election, that swap will always work out quite well for the SNP, but it could potentially lose them a few seats in the next first-past-the-post Westminster contest.  In retrospect, it seems a minor miracle that the post-indyref swing to the Tories in their former heartlands was delayed long enough for the SNP to clean up quite so comprehensively in last year's general election.

As the SNP candidate was elected on the sixth count, we're able to see where the various transfers went.  It's refreshing to discover that Scottish Labour's love affair with the Tories isn't shared by a majority of their own voters in Arbroath - 31 Labour transfers went to the SNP, and only 18 to the Tories.

Monday, November 28, 2016

Scot Goes Pop mourns the passing of Fidel Castro

We interrupt your normal programming with a special message for our resident troll, who had this to say earlier -

"Not even a mention of Fidel on this fash right wing Nat si blog."

Consider Fidel mentioned by this commie left-wing anti-YoonYoon-sis blog.

Actually, the serious answer is that the record of Castro and Cuban communism is a mixed one.  There have been appalling human rights violations, and Cuba is now basically the only outright dictatorship in the American hemisphere - but there have also been extraordinary gains in health care and literacy.  Which side of the coin matters more?  They both matter.

Sunday, November 20, 2016

To state the bleedin' obvious : yes, the US presidential polls were wrong

As GA Ponsonby wittily put it a couple of hours ago -

"The social media relationship between James Kelly and Stuart Campbell has broken down. Yet another reason for Humza to resign quite frankly."

Yes, it's true - after God knows how many years of following each other on Twitter and having a very amicable relationship, Stuart Campbell of Wings Over Scotland abruptly blocked me today.  He had become increasingly angry after I challenged his claims that the opinion polls in the US presidential election were not especially inaccurate, and he eventually told me in trademark fashion to "f*** off".  I have to say I find this a very sad development - over the years, I've backed him to the hilt over the totally unfounded allegations of misogyny and other assorted forms of bigotry, and to be fair he's also stood up for me on a number of occasions.  But throughout my near-decade of writing this blog, I've always felt very strongly that it's important never to let the 'patronage' of a leading blogger deter me in any way from pointing out when I think that person has got something wrong.  I did it with Mike Small in January.  I've actually done it a few times with Stuart before (for example in a debate about the morality of the Hiroshima bombing), and he has always previously reacted in a very constructive and mature way.  For some reason I simply don't understand, today was different.

I wasn't planning to make any further comment on the exchange, but the nature of having a dispute with someone who has several times as many followers as you do is that some of those people pile in after the event, demanding that you answer certain points.  So, free of the 140-character restraint on Twitter, here is my response.

One thing I've felt about Stuart for quite some time is that he fundamentally misunderstands the nature of the standard 3% margin of error in public opinion polls.  I seem to recall that after last year's polling disaster at the UK general election, he argued that the polls hadn't really failed, because they averaged out at a level-pegging race, which was more or less within the margin of error of the commanding 7% Tory lead we ended up with.  But that simply isn't how it's supposed to work.  The margin of error takes account of one type of error, and one type only - namely error caused by random sampling variation.  It assumes that everything else - demographic targets, weighting, etc - will be absolutely bang-on correct.  What that means in practical terms is that, although small errors in individual polls will always be commonplace, they should be distributed in a fairly random way.  If, for example, the polls had been correct to within the margin of error at the general election (leaving aside the complicating factor of a possible late swing), we might have seen a pattern in the final polls along the following lines...

Poll 1 : Tory lead of 5%
Poll 2 : Tory lead of 11%
Poll 3 : Tory lead of 5%
Poll 4 : Tory lead of 7%
Poll 5 : Tory lead of 6%
Poll 6 : Tory lead of 8%
Poll 7 : Tory lead of 3%
Poll 8 : Tory lead of 8%
Poll 9 : Tory lead of 6%
Poll 10 : Tory lead of 5%

In that hypothetical example, six out of ten polls underestimate the Tory lead, three overestimate it, and just one is absolutely correct.  That's the sort of thing that can easily happen by random chance.  But what you're NOT seeing there is every single poll underestimating the Tories, and almost all of them doing it by an amount that is either at the extreme end of the margin of error, or that exceeds the margin of error.  According to Wikipedia, and excluding a Survation poll that conveniently only appeared after the election result was already known, these were the actual last ten polls of the 2015 campaign -

Populus : Tied race
SurveyMonkey : Tory lead of 6%
Ashcroft : Tied race
Ipsos-Mori : Tory lead of 1%
YouGov : Tied race
ComRes : Tory lead of 1%
Survation : Tied race
ICM : Labour lead of 1%
ICM : Tied race
Panelbase : Labour lead of 2%

Not only did all of those polls underestimate the Tory lead, but the majority of them did so by slightly more than the margin of error.  That is not the sort of pattern that is remotely likely to occur by random chance - which tells us that the error wasn't primarily caused by the sampling variation allowed for by the margin of error, and that significant methodological mistakes were probably to blame.  (Again, that conclusion leaves aside the possibility of late swing, but it's probably correct to do so, given that YouGov's on-the-day poll was wildly inaccurate.) 

What Stuart would say, and what he effectively did say eighteen months ago, is that because on average the final polls 'only' underestimated the Tory lead by around six or seven points, they were basically accurate to within the margin of error (ie. with Labour overestimated by around 3% and the Tories underestimated by around 3%).  That just doesn't stack up.  The 3% margin of error only applies to each individual poll.  Random statistical noise should to a decent extent balance itself out over a large batch of polls, leaving you with a much smaller error.  In my hypothetical example above, the polling average underestimates the Tory lead by only 1% after rounding, which is much more typical of what you'd expect if the polls were essentially 'right'.

In my exchange with Stuart today, I was only really interested in disputing his points about margin of error, but he tried to sidetrack me into discussing other factors - in particular that polls are snapshots not predictions, and that the US presidential election is not decided by the national popular vote.  Technically, those are reasonable points to make, but when you put them together to try to construct a case that the polls didn't really get it wrong, you do start to get into the realms of the fantastical.  According to the final polls, Hillary Clinton had a national lead of around 4% going into election day.  It is simply not credible to claim that Trump could have won the election if that had actually been the result.  In any case, the possibility of a freak outcome in the electoral college is precisely what the state polls are there to warn us about - and they were even more inaccurate than the national polls!

As far as very late swing is concerned, yes, that can happen, but it won't generally be on an enormous scale, and it should show up in the exit polls (the only polls that are genuinely predictions, rather than shapshots of opinion).  As you can see HERE, the exit polls pointed to a clear Clinton victory in the electoral college.  In the vast majority of states polled, Trump was underestimated.  In almost half of the states, he was underestimated by a greater amount than the margin of error could - even theoretically - explain.

No matter how big the error, it's always possible to attempt to cobble together some kind of tenuous narrative that gives the polling firms a free pass.  If a 40% Labour lead vanishes into thin air on polling day, you can argue that 20% of voters may have changed their minds at the very last second.  But in the real world, there comes a point where you have to accept that the emperor has no clothes, and that the polls were just plain wrong.  They were wrong on Netanyahu, they were wrong on Cameron, they were wrong on Brexit, and they were wrong on Trump.  As I acknowledged the other week, everything is relative, and I would still pay much more heed to polls than to other so-called 'predictors' such as betting and financial markets.  But as of this moment, polls are plainly nowhere near as reliable as they are supposed to be.

Friday, November 18, 2016

The corner of Scotland that will be forever Trumpton

Just a quick note to let you know that I have a new article at the TalkRadio website, about the terrifying lessons Scotland has learned about Donald Trump from our own ill-fated dalliance with him a few years ago.  You can read the article HERE.

Tuesday, November 15, 2016

Winning With Willie (again)

Much of the mainstream media breathlessly and uncritically reported Willie Rennie's claim at the weekend that the Scottish Liberal Democrats were "winning again".  For those with short memories, here is how the Lib Dems actually fared at the Holyrood election in May...

* SLIPPED from fourth place to fifth, after being overtaken by the Greens for the first time.  To put that in historical perspective, in 1999, 2003 and 2007 they were within just one seat of tying the Tories for third place.

* FAILED to make any seat gains, remaining on the miserable 5-seat tally they've held since 2011.  For comparison, they won 17 seats in both 1999 and 2003, and 16 in 2007.

* SLUMPED to their lowest-ever share of the vote on the constituency ballot - just 7.8%.  For comparison, they took 14.2% in 1999, 15.3% in 2003, and 16.2% in 2007.

If all of that amounts to "winning again", the mind boggles as to what a humiliating defeat would look like.  Apparently on Planet Willie, they can tell that they're winning from the "howling" of "Cybernats".  But back in the real world, we can tell that they're losing simply by checking the results of the election.

The losing streak continued today in the Scottish Parliament as the Lib Dems grotesquely joined forces with the Tories to vote against continued Scottish membership of the single market, but mercifully were defeated by the combined forces of the SNP and Greens, with the pro-European motion being comfortably passed by 65 votes to 32.  This means that an absolute majority of Scottish parliamentarians have now explicitly backed continued single market membership, and the taking of any steps necessary to ensure that.

Labour are taking a lot of flak for abstaining on the motion, but for my money it's the Lib Dems who deserve the opprobrium on this occasion.  You can make an arguable case for abstaining on an elaborate motion that contains several different points, some of which you agree with, and some of which you don't think are ideally worded.  But for a party that was once the most passionately pro-European outfit in Scotland to actually vote against single market membership is nothing short of extraordinary.  They have abandoned all pretence of being anything other than British nationalist zealots - for them, as for the other unionist parties, it's now Brexit or Bust.

Friday, November 11, 2016

New Podcast : Trumpocalypse Now

Just a quick note to let you know that myself and Paul Kavanagh (of Wee Ginger Dug fame) are the guests on the latest edition of the Newsnet podcast, hosted as always by Derek Bateman.  Topics under discussion this week include...well, DONALD TRUMP, basically, although we do also touch on the progression of Brexit, and a possible timescale for the second independence referendum.  You can listen to the podcast HERE.

Wednesday, November 9, 2016

The neverending curse of McTernan produces its biggest catastrophe yet

I don't think what has just happened can be fairly compared to Brexit.  The prospect of losing European citizenship is thoroughly dismaying, but it doesn't scare me in the way that President-elect Trump does.  He properly terrifies me. 

I'm trying very hard to convince myself that it's not going to be so bad, and that life will go on after January.  I do draw some comfort from Craig Murray's assessment that Trump will be more of a peacemaker than Clinton would have been.  As you know, my own instinct is that the opposite is true, simply because Trump is such an unstable character.  In particular, I can't help thinking back to how he went from describing Alex Salmond as "an amazing man" to branding him as "Mad Alex" within a dizzyingly short period of time.  If his opinion of Putin were to change equally dramatically, the potential consequences for the world hardly bear thinking about.  But it's reassuring to know that at least some people have confidence that Trump will deliver what he promised about avoiding military adventurism, and we'll just have to hope they're proved correct.

If Trump can somehow avoid destroying the world, perhaps a little good may yet come out of all this, and some long-cherished goals of the left might be achieved almost by accident.  The inevitable loss of respect for America's leadership role, and the question marks that will now hang over NATO's future, might lead to the birth of a more multi-polar global order, which would be no bad thing.

As for John "The Gardener" McTernan, the extraordinary run of defeats he's helped to mastermind goes on - Gordon Brown, Julia Gillard, Jim Murphy, Owen Smith, and now the biggest casualty of all in Hillary Clinton.  It's little wonder that the media continue to worship at his altar, because there can't be a "political strategist" anywhere else on this planet who can boast a record quite like that.

Tuesday, November 8, 2016

From Brexit to Trump : can we trust the polls anymore?

I think it's probably fair to say that it's been a great many decades since US media coverage of an American election has referenced British politics so frequently.  That's a sign of how Brexit matters internationally in a way that routine changes of government don't, but it's also a sign of Trump supporters casting around for glimmers of hope wherever they can be found.  The theory is that support for Trump is vaguely similar in character to support for Brexit, and that if the latter was underestimated by the polls, there's no reason to think the former isn't being underestimated as well.

From this side of the Atlantic, there have been two reasons commonly cited for why that is probably wishful thinking.  The first is that we knew in the days prior to the EU referendum that the postal votes that had already been cast were painting a different picture from the opinion polls, and that Remain had a significant deficit to overcome on polling day itself.  On the whole, the opposite seems to be happening in the US at the moment, with the early voting data tending to look more promising for Clinton than for Trump.  The second reason is that, supposedly, the opinion polls in the EU referendum were nowhere near as inaccurate as portrayed.

The first reason makes perfect sense to me, but I have to say I think the second one is pushing it a bit.  This goes back to the meat of a rather unpleasant (and now largely deleted) argument I had with a New Statesman journalist and a few others on Twitter in August, on the topic of "can we ever trust the polls again?".  Immediately after that exchange, I had been planning to write a blogpost setting out my thoughts, but I decided against it because the whole thing had become too heated.  However, this may be a good moment to make some of the points I had been planning to make that night (but leaving aside the personalities involved, obviously).

* First of all, it really must be understood that the standard 3% margin of error in individual opinion polls does not provide any sort of alibi for the polling failure in June.  If the methodology used across the industry is basically correct, the error on the polling average should be considerably lower than 3%.  For example, if one campaign is actually on 44%, you would expect just as many polls to have that campaign one, two or three points below 44% as have them one, two or three points above 44%.  The underestimates would balance out the overestimates, and you would end up with an average that is pretty close to being bang on the money.  So it's a form of sophistry to look at the string of late polls that overestimated the Remain vote, and claim that the ones that fell within the margin of error (or came close to doing so) were all technically "accurate".

Regular readers of this blog will remember that I had been completely open to the strong possibility of a Leave victory throughout the referendum campaign, but when the last polling numbers came in on 23rd June, I finally threw my hands up in the air, and said that if the polls were right, there was clearly going to be a Remain victory of some sort.  My exact words were -

"Leave can only really win now if there's been some kind of systemic problem with the public polls - although that's scarcely unheard of."

I entirely stand by that summary, and exactly the same is true of the situation in the US right now.   Donald Trump still has a chance, but that categorically isn't because he's "within the margin of error".  He may be within the margin of error in individual polls, but if he was really tied with Clinton, and if the polls were getting it right to within the margin of error, there ought to be as many polls putting him three or four points ahead as there are putting him three or four points behind.  Self-evidently, that isn't the case.  The reason he still has a chance is because it's fairly common - as our own referendum demonstrates - for polls to be misleading due to factors that are not taken into account by the standard margin of error.  That 3% wiggle-room only allows for normal sampling variation, and basically assumes that the underlying methodology is otherwise going to be perfect - which is pretty optimistic in this day and age.

If Trump wins, or if Clinton wins much bigger than we expect her to, it'll be because the polls were wrong, just as they were on Brexit.  Not necessarily wrong by all that much, or by a historically unprecedented amount, but certainly wrong in a way that the margin of error can't account for.  (Although polling firms will doubtless attempt to make that excuse by cherry-picking individual polls.)

* It's been suggested a number of times that the EU referendum polls were much more accurate than supposed, because people tend to only look at the last batch of polls, and ignore the ones earlier in June that were more favourable for Leave.  That's plainly a load of nonsense, because the reason why the later polls moved towards Remain is remarkably simple - there was almost certainly a genuine swing towards Remain as polling day approached.

The word "accurate" is a bit slippery when used in relation to opinion polls, because strictly speaking, and with the obvious exception of exit polls, all polls are snapshots of public opinion rather than predictions of election results.  A poll can be an accurate snapshot even if it differs markedly from the final outcome.  Nevertheless, if "accurate" is used to mean closeness to the final result, it's perfectly reasonable to say that later polls should be more "accurate" than earlier ones, because the closer you get to election day, the more people have made up their minds.  Therefore, the fact that the EU polls got progressively less "accurate" towards the death of the campaign makes it worse for the polling industry, not better.  It strongly implies that there was a significant in-built error all along.  When Leave appeared to be slightly behind, they were actually slightly ahead.  When Leave appeared to be slightly ahead, they actually had a decent cushion.  And so on.

* One of the apparent saving graces for the polling industry in June was that, against all expectations, online polls proved to be somewhat more accurate than telephone polls.  Nevertheless, the performance of the online polls was significantly tarnished by a Populus poll published on referendum day that was absolutely miles out from reality - it gave Remain a 55% to 45% lead.  It was suggested to me that somehow that poll doesn't really count, because it was the only published Populus voting intention poll of the entire campaign, and is therefore difficult to put into proper context.  I must say I can't make head nor tail of that line of argument.  We know that Populus had been conducting extensive private polls throughout the campaign, meaning they'd had as much opportunity as any other firm to hone their techniques.  It may well be that a 55%-45% lead was an outlier from their normal results, but it shouldn't have happened at all if their methodology had been essentially sound.  (Even the occasional 'rogue poll' that statistically will happen one time in every twenty shouldn't really be out by as much as 7%.)

So, yes, that Populus poll does deserve to be treated as an online poll like any other, and the fact that it was one of the final polls of the campaign (when it should have been more accurate, not less) does detract from the notion that online polls in general performed tolerably well.

*  *  *

To return to the original question, I think the simplest way of putting it is this.  If you want polls to be as accurate as the industry claim them to be, then you can't and shouldn't trust them, because recent history suggests you'll often (but not always) be disappointed.  If, however, you just want a ball-park sense of public opinion that is more reliable than, say, Neil Lovatt's beloved betting and financial markets, then yes, polls are still a very useful tool, and the outcome in June bears that truth out.  It really just depends on how demanding your own expectations are.

Monday, November 7, 2016

A progressive alliance may soon be needed more than ever

Just a quick note to let you know that I have a new article at the TalkRadio website, about how last week's High Court ruling could turn out to be an emergency for pro-Europeans and progressives if it triggers an early general election.  You can read it HERE.

Saturday, November 5, 2016

Freedom to silence

I had a short Twitter exchange with the Spectator's Alex Massie earlier tonight, which was sparked by his observation that "a free press requires accepting the freedom it will write things you find distasteful or even deplorable" - a reference to the pro-Brexit media's appalling reaction to the High Court judgement. I pointed out to him that freedom of speech also requires the press accepting the freedom of others to criticise the press. He innocently denied that anyone had ever disputed that point - and so I reminded him of the furious reaction from journalists to the measured criticisms a couple of SNP politicians made of Stephen Daisley. Suddenly the goalposts shifted - somehow the criticism of Daisley wasn't legitimate after all, because that was 'beyond' criticism. It was a 'silencing'. I'm puzzled by that distinction, because although the sequence of events is hotly disputed, one fact that no-one can reasonably deny is that the Scottish Government and the SNP have no power whatever over broadcasting. Even if it's true (and it doesn't seem to be) that John Nicolson and Angus MacNeil had something to do with STV's decision to change Daisley's role, the only weapon they ever had at their disposal was the strength of their own arguments. If that alone proved sufficient, doesn't it suggest that their arguments were actually rather persuasive, and probably correct? A cynic might conclude that Massie doesn't so much have a problem with the freedom to criticise journalists, but rather with the freedom to do so convincingly and non-impotently.

The other point that occurs to me is this : if Massie thinks that the freedom to criticise journalists is not absolute, but must always fall short of his own arbitrary definition of "silencing", shouldn't exactly the same exception apply to the freedom of the press? Why should the press be allowed to intimidate judges, for example? It doesn't seem terribly outlandish to suggest that the real target of the reporting in the Daily Mail and the Sun was not so much the three High Court judges who were vilified, but rather the Supreme Court judges who will hear the appeal. The message was effectively : "you're next, unless you make a decision we approve of". Does Alex Massie think the freedom of the press extends to the right not merely to 'silence' judges, but to actually subvert the law of the land?

By the way, a little memo for the press : a direct democracy involves the electorate making decisions by referendum, and those decisions being automatically implemented by virtue of the rules laid down by the constitution. Theresa May getting to decide whether and when to invoke Article 50 is not direct democracy, any more than parliament making exactly the same decision would be. If you don't like it, campaign to change our constitution to make it more like Switzerland's. There's no point in moaning because judges refuse to ignore the law.

*  *  *

As you may already have seen, the Tories scored two local by-election gains in Aberdeenshire on Thursday.  Both were in wards where the SNP topped the popular vote last time around, although thanks to the now-familiar quirk of the STV voting system, the Inverurie result was technically a gain from the Liberal Democrats rather than the SNP.  (And I'm sure we've all noted that Mike "can't be arsed" Smithson is considerably less eager to propagandise about technicalities when it's the Lib Dems on the receiving end.)  Weirdly, the SNP vote more or less held up in Inverurie but dropped steeply in Banff - I don't know if that contrast can be wholly or partly explained by personality factors.  In both elections, though, it looks as if the Tories ultimately have unionist tactical voting to thank for their triumph (along with the traditional tendency of Tory voters to be more inclined to turn out in low-interest contests).

Inverurie and District by-election result :

Conservatives 38.8% (+21.4)
SNP 34.6% (-2.6)
Liberal Democrats 22.5% (+5.2)
Labour 4.1% (-9.1)

Banff and District by-election result :

Conservatives 44.0% (+20.9)
SNP 36.2% (-19.2)
Liberal Democrats 19.8% (+8.7)

There's actually nothing radically new in these results - they follow the same pattern as the Holyrood election in May, with heavily No-voting areas coalescing around the unionist party best-placed to beat the SNP.  To the extent that some ex-SNP voters are switching direct to the Tories, those are highly likely to be people who voted No in 2014 and who have little prospect of changing their minds at the next indyref.  What we're seeing, then, is simply a mirror-image of the phenomenon of Yes voters in working-class Scotland bringing their party allegiance into line with their constitutional preference.  That's not a cause for concern for the Yes campaign in a referendum (we're chasing floating voters, not the unpersuadables), but it might well pose a problem for the SNP in the snap general election that now seems to be a distinct possibility.

For the first time in my life, I must say that I can't muster much enthuasiasm for the prospect of an early election, because I struggle to see how it can possibly leave us in a better position than we're already in.  The likelihood is that the SNP would remain dominant, but would shed a few seats - to the Tories, and possibly to the Lib Dems as well.  The crude projections suggesting Labour could be wiped out are almost certainly wide of the mark - unionist tactical voting would once again save Ian Murray's bacon.  And the seemingly inevitable Conservative landslide at UK-wide level would rob the SNP of the balance of power they currently hold on a small number of key issues where the Tories are divided.

Would there be any advantages?  Well, it would be an opportunity for Nicola Sturgeon to further amplify the mandate she already has to hold a second referendum if she deems it to be in Scotland's interests.  And it's possible that a 1983-style crushing UK-wide defeat for Labour might lead to 'constructive despair' among the progressive unionist vote in Scotland, thus boosting support for independence.

So, yes, there are pros and cons, but on the whole I hope Theresa May continues to bottle it.

*  *  *

"But what will [the Tories] do about the Fixed Term Parliaments Act?" asks Mr Smithson in big bold letters.  Hmmm.  Would "repeal it" be too obvious an answer?  The other options would be to obtain a two-thirds majority for an election by daring Labour to vote in favour, or to deliberately lose a vote of no confidence in the government.  One way or another, though, there isn't really much of an obstacle to an election if May decides to go for it.