Dave Coplin: The Amazing Big Data Revolution

01 : 07 : 2014 Big Data : Machine Learning : Healthcare

Big data will transform the way human beings behave, communicate and interact. In some ways, it will change what it means to be human.

At the moment, big data is hard to handle. It is simply too overwhelming to be of any practical use. That will change with the next generation of intelligent machines doing the heavy lifting.

These advanced computers will be able to do more than analyse big data sets. By being able to access and connect disparate data sets and make connections between them, they will begin to provide truly innovative and disruptive ways to gain insights into human behaviour.

It won’t simply be a matter of a digital system realising that if I like Superman, I might also like Batman. The system will know whether I like the dark, brooding, brutal Batman or the slightly camp 1960s version – and change what it recommends for me next as a result.

Machine learning is beginning to automate this process, using new algorithms to leverage huge amounts of data to create new services and methods of delivery. For many sectors, this has the potential to produce nothing less than a revolution.

Take healthcare, for example. The amount of available data on people’s health could be enormous, but at present it is barely being used. Once it is made available for the kind of intuitive analysis machine learning will enable, we will have a medical system that can use correlation rather than causation to begin treating serious health problems.

As an example, Microsoft conducted some research with data in a hospital that revealed a correlation between people with mental health issues and depression and rates of recovery from heart surgery. As a result of this finding, the hospital has started designing recovery programmes aimed specifically at this group.

There was no clinical study, no peer-reviewed scientific papers. Given the results, we don’t need to know why depression causes post-surgery complications, we just know that it does.

We have been trained by hundreds of years of science to believe that we have to know why something happens. In a big data world, this matters less – and that is going to be very hard to adjust to.

The level of machine learning necessary to mine big data effectively is already becoming increasingly accessible. Many of the building blocks, such as a Cloud infrastructure, are already in place. Soon, the advanced analytics will be simplified and become available not just to big technology firms such as Google, Facebook and Microsoft, but also to the butcher’s on the high street.

The biggest barriers to this big data future are not necessarily technological, but social and political. Put simply, are we ready for a world in which our technology has this kind of insight and makes those kinds of connection?

Privacy issues are a good example. They often arise because consumers feel their data is being abused to deliver offers of goods and services they don’t want.

In time this will change. Already, a few retailers and online brands are beginning to use this data effectively to deliver a new generation of customer experiences, and in so doing are open and transparent about how they got there. As this happens more and more, people will begin to trust them further to use data to continue to make their lives better.

Dave Coplin is chief envisioning officer at Microsoft UK. His new book, The Rise of the Humans, is published by Harriman House.