What's in a question?

It's not often that I spontaneously burst out laughing when I'm travelling alone on a bus, but that's what happened yesterday when I was browsing the internet on my mobile phone, and came across the following sentence on a major media website -

"The Scottish Government agrees to change the independence referendum question, after fears that it might lead people to vote Yes."

Quite right too. Heaven forbid that people should do anything other than vote No!

I was intrigued to see the reasoning in the Electoral Commission's report, and I must say I'm not entirely reassured. In fact, I'm seething with anger on one particular point -

"There is a risk that the Scottish Government will not accept our assessment of the question and will not amend the question that it includes in the Referendum Bill. Not accepting our advice will cause controversy in the public domain...

We anticipate a high media profile and will have a suitable handling plan in place."

Did that "handling plan" include leaking in advance to the unionist press in London? Call me cynical, but I suspect it might have done somehow. Frankly, a body like the Electoral Commission shouldn't even be thinking in terms of "risk" when considering the possibility that the government might reject its advice. That's not a risk, it's a natural part of the process when you have only an advisory role, and you certainly have no business setting up "handling plans" in an attempt to head off that "risk".

In broader terms, I'm at least partly reassured that the EC didn't do what I strongly suspected they might have done, ie. 'tested' the proposed question by asking people to answer it, compared the results to two or three more unionist-oriented questions, and then plumped for the one that conveniently produced the most 'middling outcome'. However, I'm a bit startled by what they seem to have done instead, which was basically to ask people if the proposed question seemed a bit biased to them, and then take those answers at face value. Were those responses spontaneous, or were they influenced by the concerted and high-profile unionist campaign to brand the question as biased? We'll never know, but the risk that it was the latter is obvious, and that means this process cannot be regarded as entirely satisfactory.

I had naively thought that 'testing' might involve a more sophisticated approach of posing the proposed referendum question, and then asking people for the reasoning behind their answer, to see if any confusion or misdirection was evident. Instead the EC seems to have set the general public up as the experts on polling psychology, which is a bit peculiar to say the least. One question that sprung to my mind is this - were the people who claimed the proposed question was leading more likely to be Yes voters or No voters? From the vague language used in the report, the answer appears (unsurprisingly) to be that they were more likely to be No voters, and yet as far as I can see no breakdown is provided for those figures. That doesn't strike me as being good enough.

I partly agree with Marcia that the referendum question will be irrelevant, because by the time of the ballot people will be well aware of what they are voting on, and won't pay much attention to the exact wording when they enter the polling booth. But where it might make a difference is on the psychology of the campaign, which will be driven by the results of opinion polls. Respondents to those polls may not have such well-developed views, and may therefore be more susceptible to influence from a unionist-flavoured question. However, in all honesty the new question is so similar to the old one that any impact on the polls is likely to be minimal.

By the way, now that there is no doubt whatever on the wording of the question, can we expect Peter Kellner and YouGov to use it for their polls - plain, simple and unadorned? I suspect that may be too much to ask, but hope springs eternal.

(Note : I dearly wanted to call this post "That's a great question, Mr. Chairman" as a tribute to James Murdoch, but it probably would have been a bit too obscure.)