When it comes to moving a live data center that is supporting a nearly $20-billion corporation, things change daily. So even though the team overseeing EMC Corporation’s transfer of data to a new data center 600 miles away spent 12 months in a discovery process to hone our migration strategy, the process is still a bit of a moving target.
With so many moving pieces, the fact is—as the saying goes—you don’t know what you don’t know with this kind of project.
One of the challenges to keep in mind when migrating to a data center like this is that it’s not just for today. The capital investment is huge, and migrating a data center takes months of planning and many more months to implement.
Nonetheless, we have achieved great success in data migration from Westborough, MA, to our new Durham, N.C., data center in 2011. We are primed to complete the migration process by September of 2012.
The data center officially opened on September 15, 2011, with a ceremony officiated by Joe Tucci and other EMC executives and attended by a contingent of interested EMC customers. The 20,000-square-feet data center is part of EMC’s newly built-out, 450,000-square-foot facility in Durham. The remainder of the facility is lab space and EMC’s first U.S. Center of Excellence. We are also migrating data to our Hopkinton, MA enterprise data center which mirrors the Durham data in a dual-data center strategy.
The Durham data center is a fully standardized, virtualized environment, with all applications running on a single data center operating system. The data center project dates back to 2009 and the migration actually began in March of 2011.
Because we have structured the migration to minimize disruption of EMC 24/7 global operations, many employees were not even aware of some of our key moves. We migrated company email, for instance, over several weekends in October and early November with little disruption.
In one of the biggest migration events of the project, on Friday, December 2, we shut down all manufacturing infrastructure at 8 p.m., migrated ten data bases, including the Oracle production, development, test, and disaster recovery grids, and were back in operation in less than 14 hours. All the hardware was built out ahead of time to create an initial skeleton of the data bases with no data in it. We took a snapshot of the data in Westborough, moved it onto a swing VMAX array so it didn’t affect production while it was running, and then moved it over an Ethernet WAN link to Durham. We had manufacturing back up and running by 9 a.m. Saturday morning, one hour ahead of schedule.
The data migration is being overseen by EMC technical services employees along with EMC internal consultants. Project managers from EMC IT oversee the operation. Our goal for 2011 was to complete 30 percent of the migration—which we achieved with the support and cooperation of those impacted by the project.
The biggest challenge is to manage the many moving parts of such an extensive migration, from building and testing the infrastructure to staging the actual moving of the data. At the height of the build-out, we had some 30 work streams going on. With three to five migrations per quarter, we have three migration events in play at any one time.
While each migration event takes about 48 hours, the workflow prior to each event requires 10 to 15 weeks. Key to the process is creating a playbook for each application migration, mapping out who is doing what, including the technical and consulting team and the application owners.
We are using a combination of both VMware and EMC technologies in the migration process. We first considered a virtual-to-virtual migration strategy to move our hundreds of critical applications and more than six petabytes of data hosted on thousands of virtual machines. However, this proved impractical due to the large scope of the migration, the latency issues created by the 600 miles between sites, and the extended downtimes required. Instead, we are moving our applications and data over a 10-gigabits-per-second Ethernet WAN link between sites.
Bundling groups of applications and data to stage the migration over the available timeframe also proved challenging. We were initially going to structure the migration around each application. However, since our data is extensively virtualized, we found that approach wouldn’t work.
With virtualization, everything touches everything else. It’s not like the old days when a business unit acquired storage and connected it up to their own server and there were islands of compute capacity and storage capacity. Back then, moving data was far less complex. We have many more applications that now rely on fewer databases and share grid dependency. Making changes to a manufacturing application, for example, could impact an HR application.
To work within the current complexity, we bundled the migrations into 30 move events that are both application-center and database-centric.
It took many people putting in extra effort—including nights and weekends—for us to reach the current 30 percent migration milestone. But that percentage only represents part of the achievement. With all the foundation work to pre-stage for the move, we are probably more like 70 percent done with the project overall. After all, we built a data center from scratch and had it sitting there ready to accept all the new stuff.
EMCers not only realized that this migration is good for the company, but have also responded to the sense of urgency created by the impending closure of our Westborough facility at the end of 2012.
With the continued support of application owners, we will have moved some 10 petabytes of data by the end of 2012. The experience we have gained in the process will help us provide our customers and partners with first-hand insights into how to chart their journey to the cloud. It actually puts us in prime position to share what truly are the best practices with our customers
