As GA Ponsonby wittily put it a couple of hours ago -
"The social media relationship between James Kelly and Stuart Campbell has broken down. Yet another reason for Humza to resign quite frankly."
Yes, it's true - after God knows how many years of following each other on Twitter and having a very amicable relationship, Stuart Campbell of Wings Over Scotland abruptly blocked me today. He had become increasingly angry after I challenged his claims that the opinion polls in the US presidential election were not especially inaccurate, and he eventually told me in trademark fashion to "f*** off". I have to say I find this a very sad development - over the years, I've backed him to the hilt over the totally unfounded allegations of misogyny and other assorted forms of bigotry, and to be fair he's also stood up for me on a number of occasions. But throughout my near-decade of writing this blog, I've always felt very strongly that it's important never to let the 'patronage' of a leading blogger deter me in any way from pointing out when I think that person has got something wrong. I did it with Mike Small in January. I've actually done it a few times with Stuart before (for example in a debate about the morality of the Hiroshima bombing), and he has always previously reacted in a very constructive and mature way. For some reason I simply don't understand, today was different.
I wasn't planning to make any further comment on the exchange, but the nature of having a dispute with someone who has several times as many followers as you do is that some of those people pile in after the event, demanding that you answer certain points. So, free of the 140-character restraint on Twitter, here is my response.
One thing I've felt about Stuart for quite some time is that he fundamentally misunderstands the nature of the standard 3% margin of error in public opinion polls. I seem to recall that after last year's polling disaster at the UK general election, he argued that the polls hadn't really failed, because they averaged out at a level-pegging race, which was more or less within the margin of error of the commanding 7% Tory lead we ended up with. But that simply isn't how it's supposed to work. The margin of error takes account of one type of error, and one type only - namely error caused by random sampling variation. It assumes that everything else - demographic targets, weighting, etc - will be absolutely bang-on correct. What that means in practical terms is that, although small errors in individual polls will always be commonplace, they should be distributed in a fairly random way. If, for example, the polls had been correct to within the margin of error at the general election (leaving aside the complicating factor of a possible late swing), we might have seen a pattern in the final polls along the following lines...
Poll 1 : Tory lead of 5%
Poll 2 : Tory lead of 11%
Poll 3 : Tory lead of 5%
Poll 4 : Tory lead of 7%
Poll 5 : Tory lead of 6%
Poll 6 : Tory lead of 8%
Poll 7 : Tory lead of 3%
Poll 8 : Tory lead of 8%
Poll 9 : Tory lead of 6%
Poll 10 : Tory lead of 5%
In that hypothetical example, six out of ten polls underestimate the Tory lead, three overestimate it, and just one is absolutely correct. That's the sort of thing that can easily happen by random chance. But what you're NOT seeing there is every single poll underestimating the Tories, and almost all of them doing it by an amount that is either at the extreme end of the margin of error, or that exceeds the margin of error. According to Wikipedia, and excluding a Survation poll that conveniently only appeared after the election result was already known, these were the actual last ten polls of the 2015 campaign -
Populus : Tied race
SurveyMonkey : Tory lead of 6%
Ashcroft : Tied race
Ipsos-Mori : Tory lead of 1%
YouGov : Tied race
ComRes : Tory lead of 1%
Survation : Tied race
ICM : Labour lead of 1%
ICM : Tied race
Panelbase : Labour lead of 2%
Not only did all of those polls underestimate the Tory lead, but the majority of them did so by slightly more than the margin of error. That is not the sort of pattern that is remotely likely to occur by random chance - which tells us that the error wasn't primarily caused by the sampling variation allowed for by the margin of error, and that significant methodological mistakes were probably to blame. (Again, that conclusion leaves aside the possibility of late swing, but it's probably correct to do so, given that YouGov's on-the-day poll was wildly inaccurate.)
What Stuart would say, and what he effectively did say eighteen months ago, is that because on average the final polls 'only' underestimated the Tory lead by around six or seven points, they were basically accurate to within the margin of error (ie. with Labour overestimated by around 3% and the Tories underestimated by around 3%). That just doesn't stack up. The 3% margin of error only applies to each individual poll. Random statistical noise should to a decent extent balance itself out over a large batch of polls, leaving you with a much smaller error. In my hypothetical example above, the polling average underestimates the Tory lead by only 1% after rounding, which is much more typical of what you'd expect if the polls were essentially 'right'.
In my exchange with Stuart today, I was only really interested in disputing his points about margin of error, but he tried to sidetrack me into discussing other factors - in particular that polls are snapshots not predictions, and that the US presidential election is not decided by the national popular vote. Technically, those are reasonable points to make, but when you put them together to try to construct a case that the polls didn't really get it wrong, you do start to get into the realms of the fantastical. According to the final polls, Hillary Clinton had a national lead of around 4% going into election day. It is simply not credible to claim that Trump could have won the election if that had actually been the result. In any case, the possibility of a freak outcome in the electoral college is precisely what the state polls are there to warn us about - and they were even more inaccurate than the national polls!
As far as very late swing is concerned, yes, that can happen, but it won't generally be on an enormous scale, and it should show up in the exit polls (the only polls that are genuinely predictions, rather than shapshots of opinion). As you can see HERE, the exit polls pointed to a clear Clinton victory in the electoral college. In the vast majority of states polled, Trump was underestimated. In almost half of the states, he was underestimated by a greater amount than the margin of error could - even theoretically - explain.
No matter how big the error, it's always possible to attempt to cobble together some kind of tenuous narrative that gives the polling firms a free pass. If a 40% Labour lead vanishes into thin air on polling day, you can argue that 20% of voters may have changed their minds at the very last second. But in the real world, there comes a point where you have to accept that the emperor has no clothes, and that the polls were just plain wrong. They were wrong on Netanyahu, they were wrong on Cameron, they were wrong on Brexit, and they were wrong on Trump. As I acknowledged the other week, everything is relative, and I would still pay much more heed to polls than to other so-called 'predictors' such as betting and financial markets. But as of this moment, polls are plainly nowhere near as reliable as they are supposed to be.
