Plutora Blog - DevOps, Release Management, Test Environment Management
DevOps Risks: 5 Mistakes to AvoidReading time 7 minutes
DevOps arrived in large enterprises, and we’re seeing several large corporations moving to more agile software delivery and more frequent releases. We’ve helped many companies move toward DevOps practices using Plutora to provide necessary transparency to manage risk and track agile software development efforts, and over the past few years we’ve come to understand what makes a DevOps initiative succeed and what makes it fail.
In this post we focus on some of the factors that can contribute to a failed DevOps initiative – a DevOps initiative that often ends up being either abandoned or one that ultimately ends up recreating the same messy bureaucracy it was supposed to replace.
1. Fail to Define what DevOps Means to Your Organization
The term DevOps is not well defined, and you’d be hard pressed to get the same definition of “DevOps” from everyone you ask in your enterprise. Developers in your organization may equate DevOps with a specific approach to software builds and the use of popular tools such as Chef, Puppet, Jenkins, Git, and Docker. IT management might see DevOps as a continuation of existing processes with an emphasis on faster time to market and lightweight release procedures.
Without a common definition of the term you’ll have teams arguing over what is DevOps and what is not DevOps. If your software releases still involve the use of a change management tool such as BMC’s Remedy is it really DevOps? If the build takes an hour to deploy a QA build is it really DevOps? The reality of Enterprise DevOps is that every organization’s answers to these question will vary. Enterprise DevOps is compromise between self-service, rapid-fire agility and the ability to manage the risks that accompany mission-critical builds.
Some of our customers are launching rockets and running economies, “DevOps” means something very different to these clients than it means to a startup.
Before you start introducing technology and process under a DevOps initiative make sure to define a baseline for your DevOps initiative:
- What is your definition of DevOps?
- What are you trying to accomplish? And, most importantly,
- Where do you draw the boundaries between DevOps and more structured approaches to IT service management.
Fail to do this and you’ll see teams arguing over what DevOps “is.”
2. Focus on Tools and Techniques, Forgeting About People
The mistake many enterprises make is to elevate technology as the primary driver of DevOps at the expense of some of the difficult yet necessary processes that ensure that quality software that meets customer expectations. It isn’t enough to hire a few release engineers, give them a bunch of VMs, and permission to install Jenkins and Puppet. You can’t just hire a bunch of “DevOps People” put them in a room and step away expecting them to work “Magic” and make everything more efficient.
Instead what you need to ensure in any DevOps initiative is that you are taking human-driven processes in account. You need to align any existing teams such a quality assurance and release management with DevOps initiatives to avoid the common mistake of failing to adapt an existing enterprise release management process to a new approach to software delivery and service management.
You can always create more environments by throwing servers at the problem, but it’s unrealistic to expect your teams to scale overnight. Occasionally, your QA team needs to sleep.
We’ve seen a number of companies adopt DevOps, move to faster, more frequent releases driven by the needs of individual projects only to realize that an increase cadence of software delivery can lead to QA and release management burnout. If you are introducing more automation to speed time to market make sure you also think of the impact any DevOps initiative is going to have on people.
3. Ignore Governance
A lot of the rhetoric in favor of DevOps is rooted in the idea that developers need to take a proactive approach to “route around” change-averse administrators. DevOps in the enterprise tends to emerge from one or two group deciding to stage a revolution against an ineffective IT organization, and a number of us have participated in these transitions over the last decade. You work for a company that has huge system, intractable releases that take months, and, eventually, you just lose patience with the process and one or two teams decide that they are going to break the mold and move quickly.
The first DevOps were breaking the rules by design. They were comprised of independent teams “free to innovate” outside of centralized IT structures. This approach works in smaller startups, but it is a non-starter in most enterprises.
An enterprise without common standards for software architecture, release management, and environment management isn’t an enterprise at all – it’s an awful mess.
An organization with dozens of independent teams creating novel continuous deployment pipelines sounds good in a work of DevOps fiction, but it never works in practice. When we work with the largest companies in the industry they want to enable teams to work faster, but they also understand that DevOps isn’t about reducing the number of governance gates. On the contrary, if anything DevOps enables more effective, more frequent governance gates if you use a tool like Plutora to shine a light on complex release orchestration challenges.
4. Fail to Account for Risk
More frequent releases, self-service provisioning of infrastructure, infrastructure automation, continuous delivery pipelines: all of these common factors of DevOps initiatives lead to faster time to market, but at the tail end of a release process the business risks remain unchanged. Changes to production facing systems still require rigorous change management and when multiple teams feel empowered to push to production every weeks (or every day) you still need some release management function tracking conflicts and risk.
Many companies dive head first into DevOps without a full understanding of how DevOps will affect risks associated with software releases. When a company transitions from a slower, ITIL-focused process to a more agile, DevOps-focused reality release managers are often expected to “wing it” toward the end of the release cycle. They are asked to put governance gates atop a fast moving, constantly evolving process that is driven by development teams eager to release.
When organizations adopt DevOps they often lose the built-in “checks and balances” that came with ITIL. Software can be delivered faster, but the enterprise still require governance gates.
With Plutora you can let application development teams move quickly, you can allow teams to conduct multiple simultaneous release, and at the tail end of the process you can integrate with existing ITIL tools that operations expects to have in place to track and manage risk. If you just throw developers at your production servers you’ll learn first-hand how reliable production is when no one factors risk in decisions about production.
5. Run DevOps without Metrics
DevOps teams start out very eager to make large changes to an enterprise’s infrastructure and release process, but they also tend to bite off a bit more than they can chew. The smart enterprise understands that no initiative can interfere with ongoing software development and release management and they will manage the slow transition to DevOps tools and techniques over several quarters. While you DevOps teams will want to reinvent your release process overnight you release and IT managers should define metrics to evaluate whether DevOps initiatives are a success.
Enterprises expect to see hard data to back up staffing and infrastructure decisions. If you are invested in the success of a DevOps initiative make sure that you are collecting statistics that justify your investment.
If you stand up a DevOps team ask them to define roles and responsibilities. Hold them accountable for bringing greater efficiency to the organization. Regularly check in with developers and system administrators not on the team and objectively assess the results. Keep track of release and environment metrics with Plutora for teams that are involved with DevOps and teams not involved with DevOps and use data to make informed decisions to dial up or dial down particular initiatives from your DevOps teams.