The healthcare industry has faced unique challenges as a result of the global coronavirus pandemic.
For US health insurance provider, Healthfirst, it meant that their highly vulnerable and underserved customers weren’t able to visit the Community Based Offices (CBOs) as they needed. The company was already exploring how DevOps principles could help improve the products and services they provide to 1.5 million New Yorkers, but the extreme pressure the care sector was under in dealing with this extensive medical emergency, along with lockdown and social distancing measures, meant that they needed to change their model, and fast.
“Healthfirst is agile across the board for all IT work from application development, to infrastructure to operational activities. Every team has epics, features, and stories even for operational work. We are working in smaller increments and reserve capacity for unplanned activities to support production issues or changing business needs.”
Tony Mongiovi, Director of Release Management
Though agile is an established practice across the entire organization, teams are at different levels of maturity. To drive continuous improvement Plutora is providing insights into areas where teams are not consistently following Healthfirst’s organizational processes. However, while agile is an established practice across the organization, CICD pipelines are not, yet.
The requirement to provide services to customers in a remote, virtual model led Healthfirst to expedite development and deliver, at speed, a new mobile app. This was partly possible since it was the first mobile app Healthfirst had produced and was not constrained by legacy, monolithic systems that exist across the other 150 applications that have been developed over the company’s nearly 30 years of business.
The mobile application was architected for incremental change and integrated automated testing, positioning it for continuous delivery and the team currently releases at the end of every three week sprint. Since most of the other applications continue to operate in large batches of requirements and are tightly coupled with multiple dependencies they use the central release calendar to reduce release risk.
As the mobile app is able to release in their sprint cadence, they can release every three weeks, rather than the monthly calendar cadence. But the reality is that they are releasing almost two months earlier than they would, were they scheduling their releases through the calendar. This is because they don’t have to wait for the approvals and line up regression testing several months out. The team continues to follow the established Change Approval Board (CAB) process, but they are still relatively early in their DevOps journey so this doesn’t interfere with the once a sprint release cadence. As they move towards true continuous delivery, they will look for ways to automate the governance and make the CAB process increasingly lightweight, via Plutora.
As Healthfirst began to adopt DevOps ways of working, they partnered with Plutora to help them improve their release quality while accelerating deployments.
The coronavirus pandemic wasn’t only challenging for Healthfirst’s customers, but also their employees as they had to quickly adjust to remote working. No longer could they walk over to each other’s desks and check in about recent release changes or their overall status. They needed to maintain visibility and have a single source of truth in the transition to remote work. Plutora gave the teams at Healthfirst the ability to coordinate work and collaborate even during this extraordinary time.
Healthfirst intends to introduce CICD pipelines across all applications to allow them to release faster and more independently, but it’s not only the automation that needs to improve across the teams to achieve this goal without losing the quality and safety benefits that good governance brings. Healthfirst have adopted the Scaled Agile Framework (SAFe) to coordinate cross team releases and are using SAFe processes for Program Increment (PI Planning) and Agile Release Trains (ARTs) to build confidence that when a complex release happens, it works. In SAFe, an ART is a collection of value streams (anything that delivers a product or service).
Using SAFe and focusing people on planning for releases also helps drive collaborative behaviors and increased visibility into the value stream’s work and its flow. But what’s really making a difference now is the implementation of Plutora for Value Stream Management (VSM) and its release orchestration capabilities.
Healthfirst have a number of tools in their emerging DevOps toolchain including VersionOne for planning and product backlog, HP ALM for test management and ServiceNow for change management governance and service management including incident, problem and request management. Plutora manages integration between the tools and insights into the flow across them.
“Prior to Plutora, people frequently asked for more time for functional testing and they would be fire fighting. But now, I can see which pieces haven’t been accepted into the release by the respective product owners. I can then have a data driven conversation with them and find out what the problem is, allow extra time if needed or defer to a future release. Then we can go back, inspect what happened and gain learnings and insights for improvements. We still have governance and we still have oversight – but it doesn’t get in the way.”
Tony Mongiovi, Director of Release Management
Plutora has also helped the release management team throttle scope creep.
“Prior to Plutora we had a very limited view on the scope of a release. On the last day before CAB, the teams would come and ask for more time. Because the release managers are close to the testing team, they would know that their requirements had changed, but not when or how.”
Tony Mongiovi, Director of Release Management
Healthfirst has the concept of ‘scope lock’ in ServiceNow. When this is on, the release is no longer available to add change records to. However, this didn’t show anything about what was happening in the application; it only prevented new applications from being added into the Enterprise Release. Now, when a new story comes in from VersionOne, Plutora honors the scope lock status of the release in ServiceNow, blocks that change and adds a message to the record. An alert is sent to Release Managers from Plutora, which drives a conversation with the team as to what is happening and how it could potentially impact the release and determine if new scope is appropriate and ensure appropriate risk mitigation steps are put in place.
Additionally, teams frequently request extensions to release testing windows which are approved by the release team. By adding a shortlist of reasons for the time extensions in Plutora, they are now gathering data to provide insights into where particular teams are experiencing problems and work with them to resolve them. Over time, the data will show how behaviors are changing and also how the adoption of these value-stream centric and DevOps principles are impacting culture.
Plutora enables the release team to inspect the end-to-end process in SAFe, not just the release elements which has enabled them to find potential problems in the approach much earlier, further reducing the risk further down the line. For example, a flag has been set up for when the field ‘Acceptance Criteria’ does not contain sufficient detail and/or adhere to organizational SAFe process guidelines. When this field is complete, the story is accepted, but a low number of characters often indicates a poor or placeholder (e.g. “To be completed”) description of the acceptance criteria. The release train engineers look for these exceptions daily and they also provide an opportunity to have a data-driven conversation with the value stream team and improve behaviors around the Definition of Done.
Additionally, they are now able to use Plutora to provide a converged view of the complex, multiple change records that exist in ServiceNow that relate to multiple releases across all their applications. This provides complete visibility into the status of the release, something that previously several systems would have to be queried to piece together the understanding.
A ‘Readiness Dashboard’ means that the release team can now see the amount of user stories, their status, the state of the ServiceNow change record and the time until they will go to CAB in a single view; previously they would have had to reference several sources to understand this story.
The target state is for all value streams to have CICD pipelines and automate the deployments. Deployments are currently a very manual exercise although Plutora already allows the release management team to roll up all the release plans from all of the value stream teams and aggregate them. ServiceNow coordinates a few automated deployments in a number of applications today but in the future, Plutora will coordinate this activity with Healthfirst’s chosen CI server.
As Healthfirst’s teams’ DevOps capabilities improve, they plan to create a new DevOps Release Type in ServiceNow which will have a checkbox that confirms that the application is CICD ready. Being CICD ready recognizes that the team has reached an acceptable state of automated continuous testing and has achieved the levels of integrated governance that considers it continually compliant. This will trigger a different release process, decoupling the release from the monthly release calendar and driving autonomy.
Additionally, there will be an integration from Plutora to Healthfirst’s CICD pipeline that will include governance (through the release type and CICD readiness), build frequency, deployment frequency and duration metrics.
Ultimately, Healthfirst are looking forward to a target state where they are able to see the flow of work through each value stream and gain insights for further optimization whilst continuing to balance speed and risk.