At this point in the DevOps movement if you ask nearly anyone what is the hardest part of DevOps, they will invariably say changing culture. It isn’t the technology, it isn’t even process, rather it is the people that are the impediment to change. Why is this? Why is change so hard? Despite the reports and numbers, why is it so difficult to change people and transform enterprises? Before exploring how and why to change culture, let’s first take a closer look at culture and understand what it is that we are trying to change.
Culture is what you do without really thinking about it. It is a collection of behaviors, beliefs, values, and symbols that are passed along through stories, experience, and imitation. Think back to your first month at your current job or employer. How many times did you hear “…because that is how we do it?” or “…talk to Mary because she can show you how to do it”? That is culture. Culture isn’t written down. You can’t see it on an org chart. And you won’t likely find it in process flows but it is the way of life in an organization. It is the underlying, undocumented operating fabric of your organization.
For me, cultural fabrics are a lot like knit sweaters. They are designed to be resilient. You can push, pull, tug, and stretch an enterprise’s operating fabric but like a sweater, it will bounce back to its original shape. Recognizing these characteristics, you can begin to understand why it is so difficult to change and why it is so critical to dismantle the old fabric and begin to replace it with a new fabric or culture. Too often, changes are introduced without a good understanding of how they fit into the flow (or pattern) of work. They aren’t part of a holistic plan. A good example is agile development. For many enterprises, agile was rolled out to development and test teams with little thought to upstream product management practice and/or downstream operations and release management practices. MVPs and 2-week release cycles were (and are) very disruptive to organizations that typically release quarterly or less frequently. While agile was great in principle, it wasn’t embedded into a more holistic plan. While new experiences where introduced to DEV/TEST teams, product management and operations where still operating in the old model. The models were not consistent and as a result failed to achieve the expected goals and outcomes.
Many transformations struggled because they introduced new behaviors and measures that are unattainable without support and changes to upstream and downstream processes. Rather than optimizing horizontally, or optimizing a single process or activity across many products and teams, DevOps acts vertically, solving all problems end-to-end for a single application and team. This practice ensures that solutions are fully cognizant of potential up/down-stream disruptions. It stays holistic by design. The practice of building end-to-end is commonly referred to as building pipelines.
Unfortunately, this process of building a pipeline culture takes time. At first it is very slow and difficult. As more people use the new way and the new cultural fabric starts to take shape, you begin to build momentum. Over time, and with proven success, your velocity will begin to accelerate. Acceleration requires a deliberate process of intention based on scientific method. We aren’t guessing our way to success we are measuring results, radiating successes, and evolving practices, processes, and tool in the enterprise.
Measuring Results
I appreciate that defining, collecting, and analyzing metrics and data is difficult and for many painful but we need to move past this. Without metrics we cannot be successful in transforming culture. Metrics provide us the data that inform us whether the newly introduced changes are working or not. For example, if your baseline mean time to deploy was 45 days per release and your post-DevOps deploy time is 15 days. You are showing a 300% improvement in release frequency. The metric informs us that the work we did to improve collaboration, build automation, and orchestrate the pipeline is providing the desired outcomes. Without hard metrics, you would be forced to rely on anecdotal evidence and feel-good-stories from users. Metrics will help you get additional funding and support; stories will get you a pat on a back at best. Metrics are also critical in communication. Metrics are easily shared in reports, dashboards, posters, and other fun displays. Success awareness and visibility breed future success and influence late adopters.
But what should be measured?
It is going to be a little different for every organization but the process of defining metrics is the same regardless of industry type, size, etc. It all starts at the top. Every metric you create needs to be aligned with the over-arching corporate goals and objectives. These goals and objectives are where the company is planning to spend money. If you can’t figure out how to be relevant against these goals, then it will be nearly impossible to get time, funding or mind share from senior leaders and executives needed to transform your culture.
In most large enterprises, IT has already distilled these enterprise level goals into IT focused objectives. They usually look something like “drive efficiency or productivity, reduce cost, accelerate delivery, improve quality”, etc. While these are good intentions, they are not actionable metrics. To make them actionable, you need to define specific, measureable targets and then understand what process and practices will be affected to achieve those changes. Goals should be specific, measurable, attainable, realistic and timely (S.M.A.R.T). For example, a large bank wanted to improve earnings per share. IT determined that it could support this effort by improving time-to-market so it could capture revenue earlier and by reducing cost of goods. A specific goal for 2016 was to reduce deployment time from 90 days to less than 30 days. This is actionable. We know what to measure, the outcome is specific and realistic, and we know by when we need to achieve results. Beneath this targeted objective are specific team level metrics used to measure changes in behavior on a day-to-day, sprint-to-sprint, release-to-release cadence. For example, a team can measure # of code commits per week so that they can move closer to the goal of deploying every 28 days or less. If team members aren’t committing code regularly and merge activities are long and painful, then it will be nearly impossible to achieve the 28 day or less improvement target. Code commits alone aren’t going to make you successful. However, they do begin to change behaviors, aka. culture, that is better aligned with frequent deployments. Once a behavior becomes habit, you can replace the metric with something new. Just remember, it typically takes 3-6 weeks and a lot or reminding to develop a habit.
Radiating Success
Radiating change based on successes is inherent to the DevOps model and a very different approach to organizational change. While the outcome, aka. a new operating fabric, is the same, how we achieve that goal is very, very different. Unlike traditional change models, DevOps transformation opts for a “prove” then “radiate” strategy over a “distribute and inspect” model. In the traditional “distribute and inspect” model, the new operating fabric, structure, processes, and tools are defined in advance and then rolled out to teams to adopt. Centralized inspection practices, like CABs or ARBs, work to ensure people are following the new models and processes. This approach is commonly referred to as a big bang rollout. It assumes you have most of the answers up front and will only make minor tweaks as you drive adoption across the organization. The DevOps model, with “prove then radiate” approach, is much more organic. It starts with individual behaviors and uses those small changes to impact thinking and ultimately culture. It assumes that you must learn and adapt throughout the entire transformation.
The “prove then radiate” approach employs a top down strategy to help define the context, goals, and guidelines of the end-state solution however the actual solution is built using a bottoms-up approach. It starts with a single, cross-functional team and allows those closest to the problem to solve it within the context of the larger transformation. It is a decentralized model. This cross-functional team builds a continuous delivery pipeline to support a single application or component. They build an end-to-end pipeline including many of the common sticking points, like regression testing, security scans, compliance checks, and/or performance testing. Where automation tools are available or the opportunity cost of building automation is low, teams will integrate these tools into a single tool chain. Where manual tasks are required, stubs in the workflow will trigger notifications to responsible parties.
Over the course of the pilot, internal champions and SMEs are developed. Often joined with external coaches, this team supports product teams adapting and adopting the new DevOps system. It is the role of external coaches and technical SMEs continue to challenge assumptions that internal champions and experts have grown blind to. They also can provide guidance and training on new tools, techniques, and/or practices as they are introduced. Internal champions built during the Pilot stage are critical to scaling. These internal champions help navigate the organization and political structure. They make key decisions around when and how hard to push. They are the de facto leaders of change. Internal technical SMEs add credibility to the solutions. They also provide new teams with technical support and oversight. They are the hands-on experts proving that this is possible.
As the team builds a solution, they collect feedback and measure the results against the local, group, and enterprise goals. Champions share success stories of early wins that are then extended by SMEs to support dependent applications and teams. As success patterns and processes begin to emerge, new operating fabrics or cultures begin to become the standard. Over time, momentum transforms the enterprise until the old way is either non-existent or retaining a minimal footprint. This process is often referred to as bi-model IT. For additional information on how to implement change in this type of model, I recommend “the Knowing Doing Gap by Pfeffer and Sutton. It recommends the following strategies that I use regularly:
- Be clear about why? If you can’t articulate why you are doing something, then how will other people follow?
- Do more and teach more. We learn by doing and build mastery by helping others.
- Expect failure, drive out fear. Expect some work to be thrown away. Just because you are throwing it away doesn’t mean it has no value. Capture your lessons learned and don’t make the same mistake again later.
- Collaborate and listen more; compete less (internally). Internal competition is typically misused creating friction and boundaries between departments and teams. As a result time is wasted and money could be better spent to create new value. New product ideas should compete, departments should collaborate.
“Valley of Despair”
The valley of despair is a dip in adoption or momentum that occurs as all cool new ideas are confronted by the overwhelming inertia of the legacy culture. You can’t skip it, jump over it, or go around it. You need to go through it. The longer you stay in the valley, the more likely your change effort will fail. All change efforts go through the valley. The depth of dip and breadth of chasm are typically proportional to how big and revolutionary a change is. In simple terms, the valley of despair is the amount of time and effort it takes to take to transform something cool and exciting into normal and routine. Remember, you are trying to transform culture. You are trying to create new habits of success.
I liken this process to that of learning to ride a bicycle. As you start, there is nothing but sheer excitement. The training wheels are off; there is lots of support and energy from your family or friends; you are sitting on your bike smiling at the top of a small hill and start rolling down. You are riding and it is cool! This is just like building a DevOps prototype. Things are cool and exciting and you are seeing some early success.
Then you start to wobble. You lose some speed. Lose balance. And crash. It hurts – pride mostly — and you get up yelling at the bike, mad at your family or friends for making you try, and swearing you will never do it again. You are in the valley of despair. Just like your attempt to get another team to use your “new DevOps way”. If you don’t get back on the bike, you never learn to ride. Convincing you to get back on the bike and try again is hard. Just like DevOps.
It isn’t until you can pedal, balance, and steer without thinking about it that you really start enjoying riding a bike. Just like DevOps. It takes courage and perseverance to overcome the valley of despair.
Evolving the Enterprise
The most common pitfall most enterprises fall into when investing in DevOps is to recreate the same system you have today with more automation and a few people with DevOps in their title. My favorite example of this came from a key note at this year’s OpenStack Summit in Austin, the speaker talked of a company wanting to improve their release time from 44 days. They made a significant investment in cloud technologies and automation. After a number of months and a lot of money, they were able to release in 42 days. Needless to say, this wasn’t the result they were looking for. If your sponsors, champions, coaches, and SMEs aren’t willing and able to challenge the status quo and introduce new, imperfect, and at times incomplete systems, processes, and practices they will be doomed to recreate the same mess you are trying to replace. Remember the existing system is your knit sweater! It is resistant to change.
So how do you EVOLVE these systems? And more importantly, how do you build a system that can respond to change?
You iterate.
We have all heard the DevOps mantra. Start small & Build; Measure; Learn. This model it is not new. It is basic scientific method simply applied to IT and application delivery. With the exception of AWS and Sticky Notes, it is how nearly all cool things where developed. While some of the ideas might sound revolutionary like continuous delivery, the pathway to achieving DevOps-at-Scale is much more evolutionary. It starts with recognizing and committing to two things:
First, adopting a DevOps culture is big. It will and does impact nearly every facet of IT. Remember the simple goal of improving deployment frequency from 90 to 28 days. Making that shift requires both the participation and expertise from Development, QA, Operations, Infrastructure, DBAs, Enterprise Architecture, PMO, BizOps, Security, Compliance, Release Mgmt, and more. These groups need to collaborate – even when painful – to ensure that all perspectives are considered since all these team play a critical role in getting to Production. The focus needs to shift from “I did my part.” to “we have new working software running in production available to users”. Anything short of working software in production doesn’t create value for the enterprise. It is nothing more than work-in-progress. If you find yourself focused on anything short of working software in production, you are likely being constrained by “the old way”.
Second, build tool chains for change. Recognizing, that different technology stacks and applications will require different tooling to achieve DevOps or continuous delivery automation goals, the platform architecture must be extensible by design. In other words, a robust API layer must surround the workflows enabling a variety of tools and technologies to support the activities of each step or stage. For example, the build step in your DEV stage may be completed by Jenkins for open source projects and MS build for .NET projects. Both are valid tools and both need to be supported by your platform.. Forcing development teams to pick one over the other will cause productivity issues and potentially resentment in the developer community. The goal is to build tools and process where it is easier to meet expectation and harder to work around the system.
Policy and Standards
When solving complex steps like security, compliance, etc. don’t design and build for the entire enterprise application portfolio. Start with a single application. As more applications are on-boarded into the new model, look for patterns across types of applications, platforms, solutions suites, etc. Promote patterns up the policy stack to maximize reuse of associated testing frameworks and test cases. Using this model, the enterprise-wide layer should be very thin and lightweight as it needs to be relevant for ALL applications. Use feedback loops between development teams and architecture and policy teams to keep standards evergreen.
Summary
If you take nothing else from this long posting, walk away understanding that changing the enterprise is possible. It isn’t easy; there is no magic tool or wand; and many times it will hurt. Despite these and other challenges, the pathway to change is proven and using the three keys that will help accelerate your DevOps journey, namely measure your results, radiate your success, and evolve the organization.
