Measuring the predictions - The Snow In The Summer or So-So

25 August 2017
How good was Yougov's election model?

Duncan Stott asked if anyone has done a proper bit of research into Yougov's projection. "No" is the answer, even after this effort.

We've used the initial release of data from Yougov, dated 2 June. While our analysis is done with the right tool (SPSS), we're almost certainly using the wrong investigation techniques, and we've only spent an hour on this topic.

Correlations from Yougov projections to the final result:

Con .979
Lab .981
LD  .964
UKP .805
Grn .903
SNP .843
PC  .961
Oth .963

As we would expect, strong confidence in the two biggest parties, slightly weaker for the LD. Something of a hefty miss for the SNP and for UKIP.

The Yougov model seemed to cap the LD vote at about 35% in any constituency. This puts an artificial ceiling on their performance, and costs a number of seats.

For comparison, here's the correlation between 2015 vote and 2017 vote for each party:

C   .931
Lab .966
LD  .904
UKP .815
G   .768
SNP .852
PC  .965
Oth .754

Overall, Yougov has met or exceeded a hypothesis that "2015 predicts 2017 exactly".

Error watch

Yougov called the wrong winner in only 48 seats. What caused the errors? Primarily, not being held by Con or Lab; Yougov was wrong in 21/67 such cases.

Three seats went from "Con likely" to "Lab gain". Colne Valley, Warwick and Leamington, Portsmouth South. All were marked by a great increase in the Labour vote - from the LD in Portsmouth, from not voting in the other seats.

Mansfield went in the opposite direction, from "Lab likely" to "Con gain".

Orkney & Shetland is another big miss, projected as a C gain with 8% for UKIP. The result was a comfortable LD hold, with UKIP less than 2%. But it's a unique constituency, both in terms of Scotland and the entire election.

Lessons

Yougov will want to focus attention on the surprises they did predict (Lab gains in Canterbury, Kensington and Chelsea, a close race in Hastings and Rye), and not on the big errors (underestimating UKIP in Thurrock, missing the LD gain in Bath, bigging up a Ind gain in East Devon).

From these initial investigations, we reckon Yougov worked well within the Butler model. They did well where the ConLab dominated discussion and votes. In a ConLab election, this was enough to look roughly right.

We're not convinced that they fully appreciated the swing from "No vote" to Labour. It is possible that this was corrected in later releases - we are working from the 2 June release, a week before the final score.

Yougov's performance in other areas was patchy, and we have no evidence on whether their model will work with three or more major parties.

Further research is needed.

Comments? | Permanent link