Alan Kay, a renowned computer scientist said, “The best way to predict the future is to invent it.”
And this is my addition: “If we are about to predict the future, let’s use past events, learnings and data to make that prediction as accurate as possible.” This is how EMC IT came about using Big Data analytics to predict service outages.
Like many of our customers, we at EMC IT are exploring the potential of using Big Data analytics to improve the availability of mission critical IT applications and services. We also know that our customers share similar operation issues, so we are excited to share our progress as well.
What started as a pilot program to use Big Data analytics to improve the operation of our Exchange email system at EMC IT has evolved into a more extensive outage prediction tool that is piquing customers’ interest. Our Big Data Analytics for Outage Prediction system could allow our operations team to collect, analyze, store, and leverage key indicators to predict and prevent interruption in mission critical operations. It is “green fields” for PAITO (Predictive Analytics for IT Operations).
This initiative and pilot program began last April and has since enabled us to predict and prevent outages in our Exchange 2007 email system 50 percent of the time. We began with a basic concept: How can we understand what the data center is telling us? In other words, over the years, we implemented hundreds of monitoring systems, element managers and notification functions within all of the components of the data center. We’re constantly getting a stream of information about our mission-critical operations. However, in our day-to-day pace of operations, we had not taken the time to understand the correlation between all this data. We weren’t putting together the picture the virtual data center was “drawing” for us.
So we decided to use Big Data analytics to help us understand the health of our operations environment and address any problems that arise as quickly as possible. We focused our pilot on the Exchange messaging system because it is a highly visible service that is running completely in a virtualized environmennt, we have an end-to-end view of the application and we were already collecting logs and other data. We put those terabytes of data into a central container. We then engaged data scientists to create a data model and predictive analytics tools using our Greenplum-based analytics technology.
On average, our IT Service Operations Center can now predict some 50 percent of Exchange outages prior to their occurrence and take action to correct them. And since the model is self-learning, we expect our predictive abilities to continue to improve.
In talking to our customers through our IT Proven customer briefing program about our success in keeping our email up and running more efficiently, we found that many are excited about the potential of using predictive analytics to prevent operations outages. One customer put it in words better that I could ever do: “The metaphorical tear of joy was clear to see as EMC is the first organisation I’ve seen that is delivering to the data vision I have for our operations. It’s put a gush of wind in my sails and I’m very enthused to explore further…”
I am very excited to see this analytics breakthrough evolve from a concept in our own service operations to an example demonstrating the benefits of Big Data Analytics in an area that is typically under-invested. That is also a great topic for discussion with our customers when we talk about operational effectiveness and efficiency. We plan to continue to meet with customers as peers, as well as our own account managers and sales people, to help drive the development of Big Data analytics for maximum shared benefit.
After all, while some issues in EMC IT are unique to our environment, we share many of the same problems that our customers face. Through our ongoing dialogue with them, we can learn from each other and collaborate to think holistically about forging new Big Data analytic solutions.
So there it is— if we cannot invent the future, we can at least predict it!
