A slightly more detailed reply to Ballot Box Scotland on his denunciation of Find Out Now polls

The other day I posted a brief response to Allan Faulds' announcement that his website Ballot Box Scotland is basically going to treat Find Out Now polls as if they don't exist from this point on. I think it might be worth going into more depth about this, because the issues are important, and there really isn't much doubt that he's based his decision in part on a false premise.  I've checked his website and I've checked his Bluesky account (he's abandoned Twitter), and he doesn't seem to have noticed that false premise yet.

It's fair to say that Mr Faulds and I are not on each other's Christmas card lists, so it's unlikely he'll read this blogpost, and even if he hears about it he's likely to discount the contents simply because they were written by me.  But if there's any stats-minded person reading this who is on good terms with him, it might be worth gently attempting an 'intervention' without mentioning my name, because I genuinely do think he's going to damage his own credibility by "blacklisting" a substantial minority of published Scottish polls on the basis of a completely false assumption.

The simplified version of his argument for "blacklisting" Find Out Now goes like this - 

* Their poll at the weekend had the SNP at 25% of the list vote, which is implausibly low.  It's a joint record post-2021 low, when the evidence from other pollsters is that the SNP's popularity has rebounded markedly since last summer.

* Their poll had the Greens and Lib Dems on 13% apiece for the Holyrood list vote, and had the Greens on 10% of the Holyrood constituency vote, which is implausibly high.

* Their poll had Alba on 7% of the list vote, which is massively out of step with every other polling company apart from Norstat. It's an all-time high for Alba at a time when Mr Faulds thinks the party is in a very weak state (and there's a very long discussion about why he thinks they're so weak - some of which is perfectly fair comment and some isn't).

But the problem is that most of the above isn't true.  The SNP are not on 25% of the list vote in the Find Out Now poll, and nor are Alba on 7%.  The Herald (who commissioned the poll) reported the wrong numbers - it's as simple as that. If Mr Faulds looked at the correct turnout-adjusted numbers, which are available on the Find Out Now website, his mind would be set at rest to at least some extent.  The SNP are actually on around 27% of the list vote and Alba are on around 5% - probably only enough to win Alba one list seat, rather than the eight that was suggested at the weekend.  OK, that probably still understates the SNP and overstates Alba, but it's within a more realistic range.

The Greens are actually on around 9% of the constituency vote and the Lib Dems are on around 12% of the list vote - not a huge difference from the wrong numbers reported by the Herald, but again, slightly more realistic.

There is a legitimate criticism that can and should be made of Find Out Now, but it's totally different from the one Mr Faulds has made.  Where they're seemingly going wrong is by not giving their clients sufficient guidance on how to interpret the data tables.  There's not much doubt that Find Out Now regard the turnout-adjusted voting intention numbers as the definitive, headline numbers, because they've used the turnout-adjusted numbers for Westminster in their own write-up of the poll.  But they don't seem to have explained that to the Herald.  I know from my own experience of commissioning a Find Out Now poll that I was basically just handed the data tables without much in the way of explanatory notes.  I manually worked out the independence percentages with Don't Knows excluded, and I then emailed them to say something like "before publishing, can I just check I have these percentages correct?", and they said "yes".  But if I hadn't checked, I'd never have known for sure what the correct percentages were.  It's not at all hard to see how wires could be getting crossed with a newspaper client.

I also think Mr Faulds is wrong in another respect, although this is more a matter of opinion than of fact.  I don't think it's good enough for him to say "this polling company is producing numbers that don't 'feel' right to me, therefore I'm going to pretend their polls don't exist". Remember the 2017 general election campaign, when most polling firms were pointing to a Tory landslide, but Survation were suggesting a hung parliament?  Survation were relentlessly mocked, not least by Andrew Neil, who just 'knew' it was ridiculous to suggest that a radical leftie like Jeremy Corbyn could be polling strongly with the British electorate.  But it turned out that Survation were right and the others were wrong.

Demanding that polling companies must only produce results that "feel intuitively right" to commentators effectively encourages those companies to artificially "herd" their results - which in some cases will make polling averages much less accurate, not more so.

*  *  *

I launched the Scot Goes Pop fundraiser for 2025 around ten days ago, and so far the running total stands at £1251, meaning that 18% of the target of £6800 has been raised.  If you'd like to help Scot Goes Pop continue with poll analysis and truly independent political commentary for another year, donations are welcome HERE.  Direct Paypal donations can also be made - my Paypal email address is:   jkellysta@yahoo.co.uk