Big Data

The Metameme of the Internet age.


Big Data controls the way money flows, the way health care operates, the way governments rule, the way people like Obama get elected. But Big Data is even more significant than all this: it answers questions about human societies that even Adam Smith and Karl Marx failed to address. How so? Sandy Petland, pioneer in big data and computation social science, explains:

Adam Smith and Karl Marx were wrong, or at least had only half the answers. Why? Because they talked about markets and classes, but those are aggregates. They're averages.

While it may be useful to reason about the averages, social phenomena are really made up of millions of small transactions between individuals. There are patterns in those individual transactions that are not just averages, they're the things that are responsible for the flash crash and the Arab spring. You need to get down into these new patterns, these micro-patterns, because they don't just average out to the classical way of understanding society. We're entering a new era of social physics, where it's the details of all the particles—the you and me—that actually determine the outcome.

Reasoning about markets and classes may get you half of the way there, but it's this new capability of looking at the details, which is only possible through Big Data, that will give us the other 50 percent of the story. We can potentially design companies, organizations, and societies that are more fair, stable and efficient as we get to really understand human physics at this fine-grain scale. This new computational social science offers incredible possibilities.

This is the first time in human history that we have the ability to see enough about ourselves that we can hope to actually build social systems that work qualitatively better than the systems we've always had. That's a remarkable change. It's like the phase transition that happened when writing was developed or when education became ubiquitous, or perhaps when people began being tied together via the Internet.

With Big Data we can now begin to actually look at the details of social interaction and how those play out, and are no longer limited to averages like market indices or election results. This is an astounding change. The ability to see the details of the market, of political revolutions, and to be able to predict and control them is definitely a case of Promethean fire - it could be used for good or for ill, and so Big Data brings us to interesting times. We're going to end up reinventing what it means to have a human society."

While many other voices responding to Big Data are fearful, cynical, and even paranoid, Petland's perspective is refreshingly optimistic, whilst still being informed. These older, more reactionary ways of thinking about Big Data leave real, flesh and blood human beings out of the equation. What matters about Big Data is the way it demonstrates and represents deep interconnectivity. Big Data shows us the connections and relationships that lead up to major events. It shows us how all people are connected together, by social structures, by machines, by patterns in behavior and choices. To Petland, the promise of Big Data lies in this: if we can begin to understand the systems that make our technological society, then we can build systems that are better, more stable, fair and safe – financial systems that don't crash, governments that don't get mired into inaction, health systems that actually work, etc. As you can see, Petland's Big Data philosophy dares to be utopian.

For more insights into how society will be reinvented in the wake of Big Data, read the rest of Petland's remarks here.