Forget big data – it's time for big algorithms to change the world

Introduction and interconnections

It’s no good just having data – it’s what you do with it that counts. In five years, one million new devices will come online every hour, creating billions of new interconnections and relationships, and producing more and more data. But these relationships will not be driven by data, but by algorithms.

“Data is inherently dumb,” says Peter Sondergaard, senior vice president at Gartner and global head of research. “It doesn’t actually do anything unless you know how to use it, and how to act with it – algorithms are where the real value lies,” he adds. “Algorithms define action … dynamic algorithms are the core of new customer interactions.”

Forget big data, it’s time for big algorithms – and an algorithmic economy.

James Blake, CEO at unstructured data insights company Hello Soda

What is an algorithm?

Using a set of rules to follow in making calculations – algorithms – is how many of today’s most famous websites and services work their magic. Netflix recommends what you should watch by monitoring what you’ve watched, and what everyone else who has watched what you have, subsequently watched.

Confused? While the process of crowdsourcing data from all users of a service might seem complex, for a computer it’s very simple. The magic is all in the dynamic algorithm used to make calculations in the cloud.

Other examples include Amazon’s recommendation algorithm, and the Waze algorithm that directs thousands of independent cars on the road. “Products and services will be defined by the sophistication of their algorithms and services,” says Sondergaard.

“Big data is a myth – it’s about how you use the data, which comes down to algorithms and analyst scores,” says James Blake, CEO at unstructured data insights company Hello Soda. “Algorithms do break down billion of records and filter them into clear, actionable insights.” This is how we grow a truly global digital economy.

Prof. Dr. Michael Feindt, founder of Blue Yonder and creator of the NeuroBayes algorithm

Missing the point

Interconnections, relationships and algorithms are defining the future of business, not big data. “If you are still talking about big data you are missing the point, both conceptually and in terms of the opportunities,” says ex-CERN physicist Prof. Dr. Michael Feindt, founder of predictive analytics and big data firm Blue Yonder and creator of the NeuroBayes algorithm. “Everyone has big data now, but raw data on its own provides no value.”

Feindt knows what he’s talking about, having created the NeuroBayes algorithm for CERN that is used in particle accelerators to filter out interesting collisions from the non-interesting. NeuroBayes, and other algorithms, have been helping particle physicists since the 1990s, but the commercial world is only now realising their importance in business.

For instance, Feindt’s algorithm is not only still in use at CERN, but it is now helping retailers across the globe both predict and automate decisions made from data. “2016 will be the year that more and more decision-makers understand the power of the algorithm,” says Feindt, adding that the vast new tranches of data and its exchange between connected devices in the growing Internet of Things will be the catalyst for a new ‘age of the algorithm’.

“Big data is just a pile of raw material, without tools you can’t do anything – algorithms are the tools to turn the raw data deposits into knowledge gems,” says Jamie Turner, CTO of global software company PCA Predict and customer experience start-up Triggar. “Simple analytics and basic algorithms are the stone age tools of simple metal-bashing.”

The end-game for the algorithmic economy is enticing. “Truly empowered customers and businesses getting what they want, when they need it, without the delays and frustration of old world application/acquisition processes,” says Blake.

Jamie Turner, CTO of global software company PCA Predict

What is the algorithmic economy?

“The algorithmic economy includes businesses that are best at building data products,” says Sean Owen, Director of Data Science at Cloudera. “A data product gets better with use since it can learn from use – it’s not just an algorithm that makes a great data product, it’s the automation of learning and size of the feedback loop.”

However, not everyone agrees that we’ve entered some kind of age of the algorithm. “Today competitive advantage is built on data, not algorithms or technology,” says Owen. “The same ideas and tools that are available to, say, Google are freely available to everyone via open source projects like Hadoop or Google’s own TensorFlow.”

He’s right, of course – their infrastructure can be rented by the minute, and rather inexpensively, by any company in the world. But there is one difference. “Google’s data is theirs alone,” notes Owen.

Deep learning and mere semantics


Most examples of algorithms in our daily lives rely on machines interpreting data from humans. But there’s no reason why this can’t occur where both parties are machines.

“This already happens in financial markets, where programs trade with one another,” says Owen, who thinks that machine-to-machine communications are the future. “You can imagine a day when your boiler negotiates a time and price with the gas and electricity provider systems, knowing your likely morning shower time, to most cheaply ensure your morning shower is hot enough,” he says.

Sean Owen, Director of Data Science at Cloudera

What is ‘deep learning’?

In the world of algorithms there’s a lot of hype around deep learning neural networks and machine learning systems, with the goal being artificially intelligent machines that understand as well as calculate.

Cue machines that can play chess like a human, using prediction rather than calculation. “Deep learning – tools built to model nature – are more intelligent and precise,” says Turner. “These are the robots and CNC (Computer Numerical Control) machines to build something useful.”

However, there’s nothing new about deep learning. “It’s a decades-old family of ideas,” says Owen, who calls deep learning a ‘celebrity algorithm’.

“It learns its own intermediate representations of its data, which sounds like thinking,” he adds. “With some clever optimisation in the last five plus years, it’s been possible to pair it with such brute force computing that it has made amazing progress in particular areas like image recognition.”

This isn’t about new kinds of algorithms, but more about massively scaling them up – and that’s what is about to happen to the economy.

Adrian Carr, Group Vice President for Global Commercial Sales at MarkLogic

Is this just semantics?

Absolutely – the tech trends we’ve mentioned are all part of one, overriding mega-trend; technological progress that gives us a powerful insight. Should we even talk about the ‘age of the algorithms’ rather than big data?

“This is not an either/or … the age of big data has only just begun, and the Internet of Things will be the second phase of this age,” says Adrian Carr, Group Vice President for Global Commercial Sales at enterprise NoSQL platform for big data MarkLogic, who thinks both are rather remote for most businesses.

“Algorithms will provide much needed insight as to where to focus human attention … systematic algorithmic adoption is required to automate decisions, summarise information and focus attention on subjective issues,” he adds.

In terms of us humans getting more insight, the development trends in both analytics and visualisation are just as important as the scaling up of algorithms.

Better, faster

Algorithms aren’t about anything other than insight and knowledge – and both are critical for maintaining competitive advantage in the coming algorithmic economy. The potential for automation and growth is massive. In fact, it’s scary big.

“We need to be mindful of rapidly increasing inequalities – not in financial wealth, for once, but knowledge wealth and capability,” says Turner, who thinks we can forget about a 5% increase in efficiency. “These tools can make us ten times better, faster, and more agile.”

At its core, that’s what the algorithmic economy is all about; speed. “Data offers freedom and opportunities, but it can also clog the system – and this is where moving to a true algorithmic-based economy can help,” says Blake. “It keeps the world moving.”

  • Four steps for success with big data in 2015

From: Forget big data – it's time for big algorithms to change the world