Panelbase announce huge methodological changes - the new poll is not directly comparable with anything that has gone before

Well, this is a bit startling.  It turns out that the exchange/argument/debate a number of us had on the previous thread was based on an entirely false premise, and indeed on a false premise twice over.  It was certainly the case that in their last poll for the Sunday Times back in May, Panelbase continued to use a slightly different question from the one they use for their other clients - although the full question didn't appear in the datasets, we know it was different because Calum Findlay took part in the poll and quoted it in full.  But if the datasets for the new Sunday Times poll are to be believed, the question has finally been brought into line with other Panelbase polls.  So in that sense the poll is directly comparable to the last Panelbase poll for Yes Scotland - but that fact has been totally eclipsed by the revelation that two other huge methodological changes have been made, meaning that no direct comparison with any previous Panelbase poll is possible.

The first change is one that I find extremely troubling, because it could be interpreted as the first sign that Peter Kellner's attempts to browbeat his fellow pollsters into adopting a more No-friendly methodology have borne fruit.  Basically Panelbase have decided to weight their results by a mixture of how people recall voting in the European elections in May, and how they recall voting in the Holyrood election in 2011.  Because too many people in the Panelbase sample recall voting SNP in May, the effect of adding a European weighting is to increase the No lead.  The problem with this approach was covered in Survation's response to Kellner's notorious article - they pointed out that far too many people recall voting in the European elections, full stop. So either the samples in online polls are hopelessly unrepresentative of the general population (in which case no amount of weighting can be sure of correcting the problem), or else a large number of people are saying they voted in May when they didn't.  If by any chance the latter is the case, downweighting respondents who claim to have voted SNP in May could be a monumental error, because it's perfectly possible that people who falsely say they voted are disproportionately likely to also say that they voted SNP.

I can only hope that no other pollsters follow suit.  This change would only be justified if respondents' recall of whether or not they voted in May bore at least some resemblance to the actual turnout.  That is clearly not even close to being the case.

On a more positive note, the second change is one that is long-overdue across the entire polling industry - Panelbase have started weighting by country of birth.  For some reason most pollsters seem to end up with too many English-born respondents in their sample, which probably goes some way towards explaining the disconnect between canvassing returns and published polls (although it certainly can't explain all of the disparity).  So in the new poll English-born respondents have been heavily downweighted from 162 to 94, while Scottish-born respondents have been upweighted from 794 to 864.  It's possible that Panelbase are the first company to introduce this form of weighting - the only one I'm not sure about is Ipsos-Mori, who routinely ask for people's country of birth as a "demographic" question (which implies that they weight by it), and yet they still seem to end up with too many English-born people in their final results.

Panelbase imply that the two changes have effectively cancelled each other out, although I'm slightly sceptical about that.  The precise wording used is "the net effect of these two new weights is statistically insignificant", and yet it's later suggested that changes between other recent Panelbase polls have also been statistically insignificant. So it seems this definition of statistical insignificance can encompass changes of 1% or greater - which in turn implies that it's perfectly possible the No lead would not be 7% in the new poll if the old methodology was still being used. So the claims of the No campaign that their lead has increased by 4% since the last Panelbase poll should be taken with a lorry-load of salt.

I'm very grateful to Ivor Knox of Panelbase for copying me in on an email that was sent to all of the firm's recent referendum clients (presumably the Sunday Times, Yes Scotland, the SNP, Wings Over Scotland and Newsnet Scotland). The main aim of the email seems to be to head off any "conspiracy theory" that Yes would have taken the lead in the new poll if the methodology hadn't been changed - "the old weighting would also show a No lead" is the emphatic message. However, there's no additional information about whether that lead would have been as big as 7%, which takes us back to the point I made above.

What is revealed, however, is that if weighting by past vote recall had only factored in the European elections (as opposed to both the European and Holyrood elections), and if there had been no weighting by country of birth, the Yes vote after Don't Knows are excluded would have been in the region of 42-43%. That's fascinating, because it more or less explains the difference between YouGov and the other online pollsters - if Panelbase make the sort of adjustment to their past vote weighting that Peter Kellner would approve of, it's enough to take their numbers down to YouGov-type levels, whereas by the same token it's presumably the case that all YouGov would have to do to produce Panelbase/Survation-type numbers is ditch the artificial "Kellner Correction" and introduce weighting by country of birth.

Another important question is what would have happened if the only methodological change Panelbase had made was the introduction of weighting by country of birth. It clearly wouldn't have been quite enough to push Yes above 50%, but the gap would certainly have been narrower.