Last month, EMC Greenplum hosted an event at the Churchill Club in Silicon Valley to address some key questions surrounding the big data debate. The panel of passionate, distinguished practitioners included Keith Collins, Gil Elbaz, Ping Li, Anand Rajaraman and myself. Michael Chui of the McKinsey Global Institute did a fantastic job moderating the discussion.
Big Data’s Impact: Everyone agreed that big data is a paradigm shift and one of the most important forces driving the global economy today. Big data is a complex, multifaceted topic, defined by the combination of data volume, velocity and variety being generated – as such, it breaks current models of data management, experimentation, analysis and storage. More importantly, it forces us to change the way we think about data and the opportunity for new value creation.
Ping Li of Accel Partners noted that big data is a once in a generation opportunity to do things differently and create companies that are truly game changing. He assured everyone that we are in the early days of the big data transformation but also cautioned that it is an “evolve or perish” game.
Like the steam engine, electricity and computers in previous decades, big data is becoming the new and highly disruptive engine of economic, social and political progress globally. From the standpoint of competitiveness and the potential capture of value, all companies need to take big data seriously.
Data Science & The Talent Shortage: The amount of data being created and the technology around this are changing rapidly, but human skills and organizational structures are not keeping pace – and that’s a problem. Actually, the entire panel agreed that the analytic talent gap is the #1 issue C-level execs face when dealing with big data and its adoption. Organizations are leveraging big data to support all aspects of the business with predictive, real-time insight and this is driving the need for more and more data scientists. What is also changing is the notion of how we do experimentation, analysis, discovery and more importantly how we learn and make decisions. New techniques are being developed to address this and we need more people with new skills to do the work.
Innovation & Creative Destruction: Personally, I think people underestimate the degree to which big data can change their business. We are all running in a high stakes race for mainline business to stay relevant to the new, mobile/social empowered customer. This is creating an urgent demand for big data solutions, driven by a fundamental change in the relationship between companies and their customers. Overall, the panelists were optimistic and agreed there is a perfect storm in the market right now and a tremendous amount of imagination about what can be done. Globally, entrepreneurs and start-ups are working to fix that gap; VC money is flowing into the space and the demand for big-data-everything is growing the market.
In a big data world, the ability to rapidly test ideas fundamentally changes an organization’s mindset and approach to innovation. Rather than agonize for months over a choice or model hypothetical scenarios, the organization simply asks its customers and other constituents to get an answer in real time. In combination, these practices constitute a new kind of “R&D” that draws on the strengths of big data technologies to speed innovation and more importantly, create new value.
The combined effects of social and mobile technologies generating big data, surfeit raw computing power and faster experimentation will change the way vendors and end users consume information. Some really important innovations will come from putting social, mobile and geo location data to work in new ways. If you are walking down a store aisle, why aren’t all the products calling out to you encouraging you to consider a purchase or interact in a new way? Why doesn’t SIRI give you suggestions and recommendations beyond a direct answer to your question?
Predictive healthcare is another area where big data is making a significant impact. The combination of personal, social/Facebook and medical history data can have a great impact on a person’s well being as well as the quality and management of healthcare organizations. Kaggle.com is currently running a $3M analytics competition to develop a predictive algorithm that can identify patients who will be admitted to the hospital within the next year, using historical claims data.
Democratizing Data: Several other issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property and even liability will need to be addressed in a big data world. Access to data is critical – companies will increasingly need to integrate information from multiple data sources, often from third parties and the incentives have to be in place to enable this. As data becomes more open and democratized, individuals will become empowered to have a voice about how it is used, what its value is, where it goes – and this will be an information power shift like never before.
Welcome to the big data revolution!
