YouGov Distorts its Latest Election Poll to Get Headlines

With the longest British election campaign in full swing, the pollsters are getting more attention than usual. It is disappointing, then, to see a major polling company falling into bad statistical habits.

Certainty

YouGov has conducted a poll of 1800 people that reverses the Conservative lead of a few days ago, which itself reversed Labour’s lead from the previous poll. You might think that the random seeming changes in results separated by only a few days might, indeed, be random sampling errors, but YouGov is happy to imply otherwise with its headline “Labour Lead at 4%”. That headline has been repeated, without any caveats, by The Sunday Times with a banner headline of “Labour races into 4-point lead after Miliband’s TV success”. But it isn’t true.

Uncertainty

The poll was a weighted sample of 1799 people, of which about a fifth expressed no intention for the election, with 32% intending to vote Conservative and 34.5% Labour — a 3.5% lead.

Pollsters and newspapers always used to print an approximate level of uncertainty in these figures, typically plus or minus 3%, due to the fact that the polled group is just a sample of the whole population and could have differences from the average due to chance. This poll has a larger sample, so the uncertainty is 2.5%, that is, 95 times out of a hundred, the error in the poll will be less than this, which is the standard statistical test of significance for a single measurement.

Pay Attention, YouGov

But, the difference between two parties is not a single measurement, but two measurements, in this case the figure for Labour and that for the Conservatives. So the uncertainty is larger. The result is not simply twice 2.5% = 5%, since it is not likely that both measurements will be at the extremes of what could be expected. The figures should be added in quadrature (the square root of the sums of the squares, just like in Pythagoras’ Theorem), so it comes to 3.5%.

Yes, you’ve spotted it, the uncertainty is the same as the putative size of Labour’s lead over the Conservatives.

(The 26 page YouGov report on the poll fails to mention statistical uncertainty anywhere, and their website has the misleading 4% claim as the headline without the appropriate health warning. They should be ashamed to be seen as a reliable source of statistics concealing important information like this.)

Actual Result

So the actual result of the poll is “No Measurable Difference Between Leading Parties”. No, it’s not an exciting headline, and it wouldn’t be able to trigger the panic, comment and newspaper sales that the incorrect actual headline produced. But since all the true variation between Labour and the Conservatives has been obscured by the statistical noise, any other statement than “We Don’t Know Who Is In The Lead” would be dishonest.

But it is not just the newspaper at fault here this time. The 26 page YouGov report on the poll fails to mention statistical uncertainty anywhere, and their website has the misleading 4% claim as the headline without the appropriate health warning.

EDIT: The day after this poll, ComRes published a 1000 person phone survey giving the Conservatives a 4% lead over Labour, with a plus or minus 4% uncertainty. They announced:

As Parliament dissolves and the General Election campaign officially begins, the Conservatives have a four point lead over Labour in the latest ComRes poll for ITV News / Daily Mail, showing their largest lead since September 2010. The biggest change since last week has seen Labour fall by three points to 32%.

All statistically insignificant, but no indication of the fact in their press release.