Lies, damned lies, and opinion polls - The Snow In The Summer or So-So

20 July 2017
Why we're leaving the Yougov panel

To our surprise, this blog has been on the panel for pollsters Yougov since 2004. After cashing in our points, we've decided to leave the panel.

Yougov is an internet pollster, it sends us surveys every so often, asking us what we think about this and that. As an inducement to complete the survey, we're offered a token reward. The pay works out at about £3 per hour, well below minimum wage. We have a feeling that we've been asked to do more work for less pay, and that our insights are worth more than minimum wage to the clients.

The site has been plagued with technical problems, its javascript runs inconsistently and often breaks. Very often, we need to refresh a page or come back to a half-completed survey. Sometimes, we've had to abandon our work halfway - so we don't get paid, just grumpy.

The content of the surveys has been problematic. From the start, Yougov has had a slightly skeezy reputation, telling the clients "whatever you want, guv". We've come across questions that assumed outside knowledge, or that were so impenetrable that we could not understand the question. Once or twice, our honest answer was rejected as "logically impossible". We can honestly believe a product is both luxurious and cheap, but the makers do not accept this.

Yougov has built its business around a set of assumptions. The most obvious is that sex equals gender, and the only possible values are "male" and "female". This is spectacularly wrong.

Yougov also makes assumptions that are, frankly, a load of bullshit. Every month, we're invited to take part in the "Brand Index", a rolling survey that asks us about labelled products. Yougov does not consider that we may buy products for their intrinsic qualities, because we like the chocolate recipe or because that deodrant doesn't crack up our skin.

We cannot honestly give opinions about "brands" because we have no opinions about "brands". At best, a brand is a way to identify the cereal we like from the cereal we don't like. But just because we like that cereal, we won't be interested in other cereals from the same manufacturer.

The assumption is that opinion is an almost physical characteristic of the subject, which can be isolated and studied in ideal laboratory conditions. #

Are these problems specific to Yougov, or are they more general? We've gone back to the origin of opinion polls, and Herbert Blumer's reflections in 1948. Many of the points are as true today as they were seventy years ago.

More recently, Fairness and Accuracy In Reporting noted, "Those who dominate governmental decision-making and private economic activity are the main supports of the pollsters. The vital needs of these groups determine, intentionally or not, the parameters within which polls are formulated."

Yougov and its clients have certain unspoken assumptions. Foremost, they assume that capitalism is inevitable. They assume that capitalism is good, and we the people wish to help corporations make more money. These assumptions may not be true, and are never measured.

There are other philosophical problems. Opinion polls encourage knee-jerk reactions, fast responses. To get paid at 3 per hour, Yougov assumes that we'll consider questions, but won't research them in depth. The statement on screen is all we're expected to consider.

This limited stimulus is certain to be biassed. We're sure that Yougov don't intend to skew the statements, but sometimes they will get the balance wrong. For instance, we believe that the best way governments can save money is to abolish borders and let all people in. This is not an option Yougov allows us to put forward.

And so the results are misleading. And so the clients, and the politicians, are misled.

The well of democracy is poisoned. Opinion polls lead the arguments, and produce some profoundly bad results. From the limited menu, Yougov might assume that we think "immigration" is a high priority. They're right, but we balk at being lumped in with those racists in vans saying "Bugger off, foreigners".

By making decisions in a bad way, we get bad decisions.

"Americans believe the nation would be better off if its leaders paid more attention to public opinion polls, says a new public opinion poll." #

Fundamentally, we find ourselves arguing against opinion polls. They appear to be democratic but are not. We are limited to a pre-selected diet, while in a true democracy we would be able to put forward any response we wanted. The poll speciously limits our actions, the real world allows us to have two clashing ideas at the same time.

Public opinion is much greater than anything Yougov can measure. Public opinion is much greater than any binary choice.

Comments? | Permanent link