Avoid the Test Environment Management Penalty Box
Jun 1, 2015
We’ve compared release managers to air traffic controllers in previous blog posts. For test environment managers we use a different analogy, instead of air traffic controllers a test environment manager job’s is more akin to assigning and preparing gates for flights arriving to and taking off from a busy airport. An environment manager’s job is fast paced assignment and reassignment of resources in reaction to constantly changing conditions.
When projects fall behind schedule or, more rarely, when projects arrive at testing and verification well ahead of schedule, there can be delays because the necessary infrastructure and personnel are not yet available. When an organization is getting ready for a holiday season or a major project launch there are often “no gates available” for projects to use. And, if your projects are vying for limited resources you find ways to ensure that projects are ready to “board” a test environment quickly to avoid further delays.
As a test environment manager at a large organization you might not be handing out upgrades to first class, but the experience of fielding calls from a large organization of impatient users and developers can often leave you with the feeling that you are running O’Hare during the Thanksgiving travel season.
Welcome to the Test Environment Management “Penalty Box”
While the default answer to “can I have a new test environment tomorrow?” is often “No, please file a support ticket.” More often than not a test environment manager is going to do what they can to see if they can fit your project in to an existing environment. A project asks for a new test environment, and you look at a manual spreadsheet to figure out what’s available and what’s not available. Sometimes you can shuffle resources around to make it work, but often you can’t. While you can spin up a new instance of an application in the cloud, your organization is still likely bound by a real budget and test environments require a real investment of time.
The truth of the matter is that, yes, the group asking for a new QA environment could probably get a new QA system if other groups did a better job of keeping track of how they use test environments. You hand out test environments to groups every day that are in different stages of readiness. You might have a group that has a test environment reserved for two weeks that only really needs it for two days but they asked for two weeks of time to make sure they had ample time for testing.
Without Plutora you don’t have insight into what’s happening with your environments across the organization. Instead of being able to identify opportunities for test environment reclaim you just end up provisioning more hardware or telling projects they have to wait. This waiting drives up the cost of developing software as entire teams are often blocked due to test environment outages or contention. And this is pure time lost, think of a plane full of passengers on a 777 that arrives at an airport two hours early – everyone is stuck on that plane waiting for a gate assignment in a section of the taxiway known as the “penalty” box.
You can tell I spend a lot of time in airports, and this “penalty box” experience always reminds me of IT environment management. You have teams rushing to get the project complete under budget and on time only to be met with “we don’t have QA environments ready for testing.” Teams can’t move on to the next requirement or the next release because software releases usually require the full attention of entire projects. You can have teams of 30-50 being asked to just take extra long lunch breaks for a week until the environment “issue” is resolved. This is real waste and real time lost.
Test Environment Anti-pattern: Permanent Allocation
Project’s that have been frequently affected by test environment outages usually learn to ask for perpetual test environments. A high-profile website or a project for a high risk business unit will just ask for a permanent QA environment or a permanent Staging environment, and this works. It solves the problem of environment contention, but it also creates massive inefficiencies and lost opportunities for environment reclaim and reuse.
It also runs contrary to the benefits of using cloud-based infrastructure. If your non-production test environments are permanent it doesn’t encourage the dynamic approach to test environment configuration and scaling that modern DevOps should be practicing. If a group needs three QA environments and three staging test environments during a particularly busy development season then they will likely amass this many test environments until either someone self-reports that they no longer need these test environments (this never happens) or there is a critical shortage of infrastructure that prompts a test environment audit.
Development teams have been trained to “hold on” to test environments even when they don’t need them especially if the organization has a track record of being resource constrained. The ironic side-effect of this tendency to hold on to test environments even when they are not in use is that it makes the organization increasingly resource constrained while creating a situation where most test environment are underutilized.
Managing Test Environment Reservations with Plutora
It is this test test environment creep, this under-utilization of non-production test environments that drove the design of Plutora Test Environment Manager. This product was created to help companies avoid absurd situations where huge teams were being asked to wait for test environments because test environment managers lacked true insight into what current test environments were being used for. When we’ve configured test Environment Manager for companies they’ve reported immediate cost savings – projects are waiting far less than they used to without Plutora and departments have been able to quickly uncovered cases when these permanent or perpetual test environments were unnecessary.
If you suffer from “test environment creep” and if you feel like your teams are constantly in the test environment “penalty box” use Plutora to manage reservations for test environments. In Plutora when a group requests a test environment they reserve it. They make a request for a test environment with a set start and end date and they provide a business justification for an environment.
While most groups are going to end up with one or two dedicated test environments, this “reservation” approach to test environment management gives you and the teams you support greater visibility into how teams are planning to move systems through development, testing, and delivery and how this will affect your test environment needs going forward. Reservations also allow you to plan for contengiencies. What would need to be done to the reservation schedule if a project slips or is a project shows up at the gate a few hours early.
One thing is clear, no one likes being stuck on a 777 for two hours because they can’t find someone to operate jetway.
Download our free eBook
Mastering Software Delivery with Value Stream Management
Discover how to optimize your software delivery with our comprehensive eBook on Value Stream Management (VSM). Learn how top organizations streamline pipelines, enhance quality, and accelerate delivery.