Plutora Blog - Release Management
Agile Accountability: Do your Governance Gates have a Padlock or a Warning Sign?Reading time 6 minutes
Most large software enterprises are facing a tough choice: on one hand you have the control and stability of traditional ITSM, and on the other you have the self-service freedom and agility of more modern approaches such as DevOps. As organizations move toward self-service releases and more rapid iterations change management presents a problematic bottleneck for teams looking to move quickly.
In this post I discuss an example from one of our customers and how they’ve used Plutora to change the way change management works to adapt to an increasingly agile reality.
A few months ago I was walking one of our customers through Plutora’s ability to model governance gates in a release. Our tool gives a release manager the ability to define checkpoints during a release cycle. For example, a project may require formal sign-off from QA in order to proceed, or a release may need explicit approval from a specific manager in a change request filed in BMC Remedy. This is the reality in most large companies – you can be on a conference call with 50 people, everyone is ready to go, but the whole team has to wait for someone to click on the “Approve” button. Governance gates are very real for large systems, and they’ve traditionally had a strong “padlock” on production deployments.
“True Believers” in Change Control
This customer is a “true believer” in change control. I use the word “believer” on purpose. When it comes to change control you have two groups: people who view it as a laborious, often unnecessary process, and people who believe it to be an essential control over the chaos of software development.
To a true believer Change Management is required for any change to a production-facing system. No matter how small, any change has to have a Change Request (CRQ) in BMC’s Remedy Change Management System (CMS). This change must have an accurate assessment of the level of risk, a detailed accounting of systems impacted, and a long audit trail of approvals.
The Overriding Imperative: Move Faster… Yesterday
While we were discussing his organization’s change control process he said something interesting:
“It isn’t like someone couldn’t just skip the CRQ and deploy directly production. Unfortunately, this is happening all the time with our new DevOps teams. We’re trying to figure out how to unblock teams but still maintain control.”
What struck me was the contrast between his organization’s commitments to change management. His is the kind of company that manages real release risk, and his change management team is admitting that the existing approaches to change management don’t work. They don’t align with DevOps or agile, and then breakdown as teams accelerate toward weekly and daily release cycles.
Your Daily Release Needs to be Submitted to the CAB Four Days in Advance?
While most of his projects continue to follow the established process, he’s in the process of perfecting a new approach to change management that puts the emphasis on accountability over control. Plutora’s a part of this solution.
With the introduction of DevOps operational responsibility has been shifted to individual teams. Teams have the ability to push releases to production directly. In the past, this would have been unheard of. Development teams would have thrown software over the wall that separates development and operations and the process of verifying and approving the release would begin – this was back when software was released once every two months so change control had days of lead time to discuss an upcoming release.
As teams assumed more responsibility and as timelines accelerated change management was quickly becoming overwhelmed with requests from frustrated development groups. In one example, a team was ready to deploy to production every single day. Even though they were ready for a daily release cadence, change control continued to insist on four days of notice for each release. Management stepped in and told change management to move faster.
Unlock the Gates, Get Some Warning Signs, and Install Cameras
This customer’s solution was more freedom coupled with greater accountability. If teams want to deploy as quickly as they can write code, let them do so and design systems that let change management hold teams accountable for compliance and quality. Let developers move as fast as they need to and automate the creation of CRQs from change sets generated by developer-facing tools.
Instead of presenting developers with a locked governance gate he’s removed the padlock and installed some serious warning signs telling teams that production deployments are monitored and audited by change management. Has some bad code made its way into production since the change? Yes, but he’s also realized that bad code was making it past his group in the past. The difference now is that he’s used more automation to create more accurate CRQs when in the past he’d rely on developers to tell him what was in a release.
Plutora’s Automation Changes Change Management
Plutora supports this process by automating the creation of a CRQ in Remedy from systems like JIRA. When one of his development teams is ready to run a release they can trigger the CRQ creation at the click of a button and his department is immediately made aware of the risk level and the changes involved in a release.
Developers are warned about the process and the necessary approvals ahead of time, and detailed records are kept of which teams pushed what changes when. If a team is consistently introducing bugs into production or if a team is a constant source of change-related downtime. His group is now responsible for tracking changes and holding teams accountable. There are still some high-profile systems requiring a full signoff that won’t be automated in the short-term, but many of his teams are transitioning to this new model.
Moving from a command-and-control paradigm for change manage toward a more decentralized approach while preserving the ability to manage change is something that all industries will need to do as software delivery accelerates. Plutora is designed to support both the existing, locked governance gates alongside newer models that emphasize agile accountability. It provides this support because we’ve designed it to integrate with just about everything.