Continuous Improvement in a CI/CD Pipeline

In 1993, the movie “Groundhog Day,” starring Bill Murray, launched its way into our hearts and minds as one of the top-rated movies of all time. In the movie, Phil Connors, played by Bill Murray, comically lives the same day over and over again. His objective, as he repeats each day, is to learn step by step how to improve himself and reach his goal.  While Phil’s attempts to live the perfect day take many twists and turns, he does eventually reach his goal.

This movie is a good example of Continuous Improvement. Phil lived each day with the objective to achieve a state of perfection in each and every moment throughout the day.  Similarly, continuous improvement is about iterative improvements to the day-to-day effort to improve the product, service, delivery and customer experience.

To avoid getting too broad in our discussion, we are going to talk about continuous improvement as it relates specifically to the process of software development.

What is Continuous Improvement?

Continuous improvement is a philosophy with the objective of creating a culture that allows anyone on the team to make or suggest improvements to a product or process at any given time. So, whether you accept and implement improvement suggestions immediately, or if you do it at the end of each sprint, the key is to accept input from everyone on the team and to make ongoing iterative improvements to the development process, workflow, toolset, etc. “Continuous Delivery,” written by Jez Humble and David Farley, states:

“The whole team should regularly gather together and hold a retrospective on the delivery process. This means that the team should reflect on what has gone well and what has gone badly, and discuss ideas on how to improve things. Somebody should be nominated to own each idea and ensure that it is acted upon. Then the next time the team gathers, they should report back on what happened.”

The continuous improvement cycle includes four key steps:

1. Plan & Measure

2. Develop & Test

3. Release & Deploy

4. Track & Fine Tune

While it might seem that a culture of continuous improvement is something that can easily be achieved by holding a few improvement meetings and saying it exists, you’ll quickly find that continuous improvement is not a destination, but a journey. In the ideal scenario, the empowering of all team members to improve any part of the process and product any time will permeate the company culture.

It’s important to note that continuous improvement doesn’t call for the wild west mentality of abandoning all process checks and balances. Depending on the type of improvement, the individual who has the initial idea should still follow the appropriate approval processes. For example, if it impacts only their own personal processes, a person may be sufficiently empowered to implement the changes directly and immediately.  However, if the proposed improvement involves others, they would need to submit the suggestion for team discussion, or submit it through the appropriate channels for analysis, approvals, and implementation.  The key point is that these proposed improvements can be submitted by anyone, at any time, and they aren’t necessarily dependent on a scheduled meeting or event.

Example of Continuous Improvement

A few years ago, a multi-national financial institution had developed a new online product that enabled customers to quickly get approvals for auto loans, which greatly simplified the car purchasing process. Being a one-of-a-kind service at the time, there was a lot of trial and error involved. The technical project leader established a continuous improvement culture that empowered everyone on the team to look for and propose ways to make improvements to any part of the technology or process at any given time. They no longer had to wait for a weekly meeting to make suggestions. Once approved, they could implement changes and improvements as quickly as resources became available to do so.

This had a powerful impact on team morale. Everyone felt empowered and valued as a contributing partner. It improved the efficiency of the online tools, workflows, behind-the- scenes information processing, approvals and much more. Delivery speeds and overall product quality increased across the board exponentially. The standards for nearly every KPI across the board were being exceeded and rewritten on a regular basis in order to keep up with the accelerated workflow, delivery velocity, and product quality.

These improvements in product delivery, customer satisfaction, and overall quality also got the attention of others and became the primary selling point for partnered alignments with several key vendors that they had been working on for quite some time

Origins of Continuous Improvement

The roots of continuous improvement can be traced back to Toyota Motor Corporation in the early 1990s. Leadership had a goal to make the product and the manufacturing process the very best it could be. They recognized that the leadership staff alone was not sufficient to come up with every idea needed to make the process better. They needed the help of the workers on the assembly line, the office staff, and even the janitors. They empowered everyone with the ability to make improvements at any time and, if necessary, to even stop the assembly line to allow for corrections and improvements to both the product and the process.

While it seems counter-intuitive to empower everyone with the ability to stop the assembly line, what they quickly found out is that the process was greatly improved. Both the product quality and manufacturing speed saw huge improvements, not to mention the many processes behind the scenes.

Impressed by the efficiency and repeatability of their processes, other western organizations took notice and performed several landmark studies on the success of Toyota Motor Corporation. Many people claim this is where the Lean methodology originated. We’ll talk more about Lean later.

Continuous Improvement vs. Value Stream Mapping

While continuous improvement is often considered to be similar to, or even the same as VSM, it’s actually not the same at all.

VSM is a very important activity that every development team should do on a continual process to better understand the value stream and their role in the process. This is something that can be immensely important to build and strengthen cross-team relationships and collaboration. For more information on VSM, go the article titled “Value Stream Mapping.”

Continuous Improvement vs. Continual Improvement

Despite sounding nearly identical, and having a seemingly similar meaning, the differences between the two are important. Merriam-Webster defines the word “continual” as “Recurring in steady, usually rapid succession.” This is the meaning that most industry experts refer to when talking about continual improvement, though many people use these terms interchangeably.Continual improvement refers to the efforts of improvements that are limited to a set or fixed cycle or a recurring basis. An example of continual improvement would be a Quality Control meeting, scheduled on a weekly or monthly basis, to discuss product quality improvements. These types of meetings are very focused and limited in scope on a specific segment or product element. The suggestions for delivery improvement are also typically only accepted during the meeting.Continuous improvement, on the other hand, is a developed culture that empowers anyone on the team to propose and develop improvement solutions, often without having to wait for a scheduled meeting to propose it, plan it and then act on it. The methodology of continuous improvement has no defined scope. If a team member has an idea for improvement, depending on the defined process in their organization, they can either address it immediately or submit it through the appropriate channels.

Kaizen

The term “continuous improvement” is often used somewhat interchangeably with “Kaizen.” Kaizen is a Japanese word for “improvement.” It’s derived from two separate words, Japanese words “Kai” and “Zen,” which have the literal translation of “Change Good,” or “Change for Good.” This refers to activities that continuously improve all functions, processes, and products, and include all employees, regardless of position or title.

For software development teams, the application of Kaizen methodology would be the development of a culture to seek for improvements at any point in the software delivery lifecycle, the product itself, the collaboration process, the management process, the tracking process, etc.

The phrase, “a Kaizen event,” is often used in reference to a scheduled meeting or activity, such as a value stream mapping activity in which improvements are proposed, discussed, approved and implemented. However, having these types of regularly scheduled meetings, with a fixed focus of activities, doesn’t necessarily mean you have a continuous improvement culture.

Lean

The Lean methodology creates a focus of improvement on eliminating waste and improving the product for the customer. The definition of waste can be in the form of actual materials, lost time, inefficient processes, and much more. In value stream mapping terms, it’s anything that doesn’t add value to the product.

Lean is a very common methodology that has permeated the thought process of nearly every type of industry in one form or another. For additional information about Lean, there is an excellent article called “5 Lean Principals every Engineer Should Know,” written by Mark Crawford.

Continuous Improvement in a CI/CD pipeline

The sprints and user stories found in CI/CD environment are much smaller and are developed in rapid iterations using small, more manageable code segments. Though the theoretical application of continuous improvement in a CI/CD environment would be somewhat similar to that found in a waterfall environment, the practical results would be much different. Team members in a CI/CD environment would be looking for ways to improve things like user story or requirements tracking, the code check-in process, unit testing, automated build processes, test environment management or automated deployments, in rapid succession for each sprint or story. Team members of a CI/CD environment are also typically more directly involved or even cross-trained in other functions of the development lifecycle than those team members of a waterfall methodology who are either developers or testers, but not both.

Implementing a Continuous Improvement culture

Developing a culture of continuous improvement is not going to be established in one meeting. Here are our eight key components to establishing a sustainable continuous improvement culture.

1. Supportive and Engaged Leadership – As with any initiative of this type, the leadership has to be firmly committed to the changes. They need to be supportive, open and willing to consider every single idea even if it doesn’t initially seem like a good idea. They also need to make sure the team members feel it’s safe and un-threatened when they pitch their ideas. If not, it will be the last and only idea you ever get from them, and the end of your culture

2. Engaged and Empowered Team Members – This is the most important component. If your team members don’t feel that they are empowered, or are disengaged, they won’t bother approaching leadership with any ideas for improvement and that will be the end of your continuous improvement culture. It’s a good thing to “go to Gemba,” but it’s infinitely better to get unfiltered input directly from those who live it and know what type of things will really make a difference in the day-to-day work.

3. Consistency – If the wind of change blows this way or that, the team members will never know what the key objectives really are and will be continually second-guessing themselves. Over time, this will result in an every-man-for-themselves mentality taking over the team. It is important to clearly establish the workflow, gates, and Even if there are ongoing improvements, iterative improvements are acceptable, but a series of flash-in-the-pan efforts will never gain team trust and culture momentum.

4. Simplified and Always Available Idea Intake – if you’re going to accept input from your team, you need to make it easy for them to submit the ideas. If it’s a difficult process, or only available during certain windows that might be difficult for any of the team members, your input and resulting improvements will be greatly restricted.

5. Adaptive and Enabling Technology – As with any methodology, the team needs to have a toolset that supports what needs to be done. Continuous improvement is no exception. Your toolset will need to be one that supports the occasional, or even frequent, change to workflow, policy, gate management, etc. At the same time, your tools need to be easy to use and non-restrictive for the team to access and use.

6. Trackable Results with Visibility – It’s important to have an established baseline against which you can measure your improvements. This will enable you, and your team, to know if you are moving in the right direction, or if a given “improvement” needs to be backed out. You need a solution that provides automated performance tracking so that you aren’t reliant on incomplete or error-prone manually entered data. These results also need to be accessible and visible to both the team members and any stakeholders.

7. Keep it Simple - If you want a process or culture to be sustainable, it has to be kept as simple as possible. If not, it will be dead before it comes out of the gate. In the effort to keep it simple and usable, you should also ask for and take input from team members on how to improve the continuous improvement process.

8. Keep it Alive – Nothing is more frustrating to the morale of the team than to have a series of flash-in-the-pan initiatives that are all the rage for a week or two, but then are never heard of again. Don’t make this the flavor of the day. It needs to be regularly discussed, taught, reinforced, encouraged and rewarded.

Continuous Improvement Tools

There are a number of tools that are used throughout the application delivery process. The challenge is to find a solution that can work with each of those tools that you already use in your organization, without complicating or breaking the process and workflow.

This is where tools like Plutora are ideal. Plutora integrates with hundreds of existing toolsets and provides a collaborative platform that tracks and shares information, not only from one toolset to another, but throughout the entire organization. It’s a delivery management platform that improves the management and visibility of the entire value stream with automated tracking of requirements, sprints, user stories, and projects. It also provides a configurable workflow and gate management that can be adapted as needed to address ongoing improvements.

One of the key advantages of Plutora is its powerful built-in reporting and analytics features that provide visibility throughout the organization’s past, present and future delivery performances.

Reaching the Finish Line…or Not.

As was mentioned at the beginning of this article, continuous improvement is not a destination, but a journey. If done correctly, it will become a culture that, much like the development delivery process, will continue to be improved upon month after month and year after year.

“Success is never final. Failure is never fatal.”-Author Unknown