Tagged: information theory

Seeing Like A Database: the problems with big data

“Big data” has been one of the buzzwords of 2011, and grand claims are being made for its power:

The world is becoming data-ized as digital information and numerical measurement is being applied to all aspects of what people do, particularly things that couldn’t be measured before because it was impractical or impossible. (Think: using wireless and GPS in cars to base insurance premiums on where and when people actually drive, as has been possible since 2007.)

The impact will be as profound as the scientific method in the 18th century — which quickly moved past the sciences and left its mark on all areas of human endeavor. For instance, what is “quantitative decision making” in management, if not the scientific method applied to business…. Likewise, the BigData revolution is plowing through the sciences, and also jumped into mainstream areas, such as business and government.

Data; boring but… by Ken Cukier, 6 March 2011

The problem with these claims is that they conflate increased power to capture and store data with (i) being able to extract meaningful insights from it, and (ii) being able to successfully act on and implement these insights, with (iii) no unexpected or adverse effects. Clearly cracking the first part doesn’t save the world on its own.

Further, big data evangelism often trips over into technocratic thinking, a belief that ‘nerdpower makes right’. Excitable blogposts about exabyte datasets, rather than defining the right problems to solve. Wide-eyed admiration for the amount of data that can be gathered, without recognition for the ethical rightness (or otherwise) of doing so.

Which is to say that big data is fundamentally political. Whether we choose to theorise it as technology or knowledge [actually, there's a good PhD proposal...], the act of recording the world in this way privileges particular values, worldviews and types of action.

In his blog post Lessons of the Victorian data revolution, Pete Warden insightfully makes the connection with technocratic thinking and brings in that great study of central planning, James Scott’s Seeing Like A State:

James Scott’s “Seeing Like a State” looks at the legacy of the Victorian scientific revolution, and shows how the very success of its ideas had a dark side. [Similarly,] Creating datasets may help technical people [...] to understand problems and propose solutions, but it also means that [...] other people with deep, lived experience of the domains will be overruled. In the 20th century the prestige of the scientific toolkit was used to justify disasters like the collectivization of agriculture, as technocrats around the world wielded numbers to take power away from “inefficient” smallholders. Those figures were mostly proven bogus by reality, as plans with no knowledge of conditions on the ground failed when confronted with the wildly variable conditions of soil, weather and pests that farmers had spent a lifetime learning to cope with.

Lessons of the Victorian data revolution by Pete Warden

If you’ve not read ‘Seeing Like A State’, incidentally, I recommend it. In it Scott surveys the great utopian schemes of the 20th century, from Le Corbusier’s urban planning in Brasilia to Russian collectivisation of agriculture and China’s Great Leap Forward. Each well-intentioned and yet spectacular failures, with millions of deaths. His argument is that centrally-managed planning does not work because it rides roughshod over the complex interdependencies on the ground.

Perhaps, under ‘big data’ ideology, we might ask – is this not simply a problem of too little information? We have the capacity to measure everything now – did Corbusier or Stalin fail because their data was not sufficiently granular?

Scott would disagree. The problem at hand is not quantity of knowledge but its very type. Common to each central planning disaster is a belief in a high-modernist ideology claiming that science can improve every aspect of human life, and an authoritarian central power willing to effect large-scale re-orderings of society and nature. “Big data will solve everything” can, clearly enough, be another iteration of the same. Scott – and, in fact, Friedrich Hayek’s criticism of centrally-planned economies (The Rule of Serfdom, 1944) – is that this disregards local and personal knowledge (Scott might add, embodied and tacit knowledge), and the complex diversity of organisation required and ends sought. (Hayek may believe that this can be summarised through the price mechanism, but Scott’s metis (local knowledge) is rather less reducible than that.)

Central planning – or big data – may seek to make the complexity of local situations legible to systemised, technocratic thinking – but the two are essentially incommensurable. Talk of ‘big data’ needs to be visible as something bringing with it a particular modernist worldview, and alongside that a particular relationship of power over the specificities – places, people – represented as nodes and datapoints. Technology is rarely value-neutral.

*

This is not to say, however, that ‘big data’ is necessarily socially oppressive. Perhaps there are alternatives – I am still thinking this through.

In my recent Bugged Planet post, I drew attention to Indy Johar’s tweet where he noted “the asymmetry of personal data, open for the 99% & deep analytics for the 1%” [source]. This raises the question, what if the analytics were open to the 99%? What would this take, what would this look like, and would it actually redistribute power in any way?

Macro trends: black box algorithms & the end of the middle class

A couple of days ago I was thinking through the question, what’s important now? Or rather, what is now – what are the currents shaping the way the world is going over the next 10 or so years?

Perhaps this was inspired by Jon Henley’s article on September 11th, which argued that it wasn’t actually the “day that changed everything”, and many of the geopolitical events seen as consequences of the attack may have happened regardless.

One key trend is clearly black box algorithms:

In a speech at the technology conference TEDGlobal this summer, computer scientist Kevin Slavin argued that a profound shift is taking place: maths is undergoing a “transition from being something that we extract and derive from the world to something that actually starts to shape it”.

The maths Slavin is talking about, and Harris is writing about, is algorithms. We are, he says, living in an “algo-world”. If Slavin is right, algorithms are shaping everything from the goods we buy to the value of the money with which we buy them.

[...]

The thing is, as systems of algorithms get more complex and take control of ever greater areas of everyday life, concerns are being raised over how much we’re able to track what they’re up to. The answer is: not all that much. As Slavin puts it: “We’re writing these things that we can no longer read.”

[Welcome to the algoworld, Sam Leith, Evening Standard, 12th Sept 2011]

Second, and bigger point – the stagnation and decline of the middle class standard of life.

Take a story that appeared in the Wall Street Journal Monday. The tale is nominally one about marketing strategy and it looks at how giant firm Procter & Gamble sells its household goods to its customers. But the picture that emerges is terrifying. P&G, it transpires, is cutting back on marketing to the disappearing middle classes, instead selling more and more to either high-income or low-income customers and abandoning the middle. Other big firms, like Heinz, are following suit. The piece reveals there is even a word for this strategy, helpfully coined by Citibank: the Consumer Hourglass Theory – because it denotes a society that bulges at the top and bottom and is squeezed in the middle.

The story contains some scary figures, such as the fact that the net worth of the middle fifth of American households has plunged by 26% in the last two years. Or that the income of the median American family, adjusted for inflation, is lower now than in 1998.

Or look at a story in the New York Times Tuesday. It starkly shows how the plight of the American working person has worsened. Solid jobs that once provided a secure grasp on middle class aims (a house, college for the kids, a retirement) have changed to become low-wage ones. It looks at the situation of some Detroit auto-workers, pointing out that new hires can find themselves working opposite long-term colleagues who do similar jobs yet earn twice as much. The system is called a “two tier” wage structure.

Perhaps that system can be justified as an emergency measure to keep Detroit’s auto-industry alive and help it survive the current tough times. But, like the Consumer Hourglass Theory, it actually looks far more like the permanent shape of things to come. American society is bifurcating, squeezing the middle class out of existence. The ranks of the poor and low-income earners are growing and the rich are doing just fine – and no one is talking about it, much less doing anything about it.

[The decline and fall of the American middle class, Paul Harris, Guardian, 13th Sept 2011]

More thinking on these later…