How to Clean Up Polluted Test Environments in 9 Steps

Take thousands of developers, distribute them over several departments, and split them up into small teams of 10-20. Now take these 20-50 development groups and ask them to create a distributed, service-oriented architecture. Without a solid approach to test environment management your organization will end up with a real “mess” of testing environments.

Test environments might start off nicely organized, but as teams come and go and systems mature and change over time. You’ll end up with test environment interdependencies that look like a bowl of spaghetti. How do you clean this mess up? Where do you even begin? In this post, I’m going to outline a series of steps you can take to clean up your polluted test environments.

Step #1: Create an Inventory

The first step in cleaning up your test environments is to understand the scope of your problem. How many test environments do you have? How often are they updated? And, what is the process for keeping track of requests for new environments?If your organization has experienced rapid growth over the past several years, or if you work at a company that has expanded rapidly through acquisitions you might not have a process defined, and you might have several competing ways to keep track. Don’t just create an inventory in a static Excel spreadsheet that will be outdated the second you’ve saved it. Use Plutora and start keeping track of environments and how they relate to your application development teams and the releases they conduct.

Step #2: Define Test Environment Management for Your Organization

When you are a test environment manager you often feel like you have twenty different bosses each with a different idea of what it is that you do. Some of your application owners expect you to configure and deploy applications to test environments while others are just looking for a few VMs so they can take it from there.

Cut through this confusion and define the purpose and objectives of your test environment management function. Different organizations have different requirements depending on the capability of teams and the applications being supported, but you need to communicate with all of your customers. What services does test environment management provide. Who is responsible for VM creation? Who is responsible for release management and deployment? Answer these questions as part of your cleanup effort.

Step #3: Assign Responsibility

RACI diagrams are a necessity. For each of your test environments record who is responsible and accountable. Then keep track of who needs to be communicated with and who needs to be kept informed of changes. Plutora is the tool that allows you to associate a RACI matrix with every test environment so that you can ensure no environment will fall through the cracks and become another, polluted and unused system.

Step #4: Isolate Your Networks!

One of the most prevalent and dangerous practices in the industry is mixed test environments. This is when a QA or a Staging system interacts with production services and databases. These mixed environments occur when test environment managers run out of resources and teams are forced to piece together testing systems that span multiple environments.

Avoid this by asking your infrastructure teams to segment your testing and QA networks from production. You never want to connect a QA system to production, and with environment pollution this practice becomes dangerous because you can start to lose track of the boundaries between QA, Staging, and Production. Isolate your QA, Staging, and Production networks and you can start to clean up polluted test environments without having to worry about production impact.

Step #5: Establish a Common Policy for Application Owners

The most difficult part of being a test environment manager is responding to an almost constant request for environments. Everyone will always need more QA and staging environments, and sometimes you have to say no. If you have a defined policy for resource allocation it will make these conversations easier. Define a set of expectations for application owners. How many QA environments can you make available to a development group? How can they justify requests for more environments? What is your exception policy for test environments if someone needs a temporary allocation of excess capacity?

It’s often easy to say yes to every request that comes your way, but if you do this you’ll find yourself running out of hardware for other projects. A policy for test environments helps you control your spend on hardware, and it also helps avoid situations where application owners feel free to ask for an unreasonable number of testing environments.

Step #6: Divide and Conquer

If you work at a large company with thousands of developers you likely have 20-50 internal customers. Don’t treat each of them as a unique customer as this approach rarely scales. Try to group them into departmental or functional groups. You might have 10 customers all working on the same type of application in the same department. If this is the case, group them together and assign a single individual as your point of contact for these releases.

You may also have many, cross-functional teams that all release at the same time. They are working on different technologies, but they will always be coordinated to perform releases on the same product. Or, you might have several independent teams all sharing the same technology platforms. Find commonalities and intersections across your customers and look for ways to streamline the process for everyone.

Step #7: Tame Your Testing Data

Test environment management is made easier when you have an automated approach to creating test data. In fact, if you don’t have a good approach to test data and data masking it makes it much more difficult to achieve the sort of agile approach to test environments management that development teams expect.

Don’t just focus on deployment automation and efforts to make your application servers more agile. Involve your DBAs in the conversation surrounding test environment management. Make sure that they understand the importance of keeping up with advancements in infrastructure automation, and don’t take on the task of test environment database administration as part of test environment management. Use Plutora to identify just how much support test environments need from DBAs and use the tool to justify resource allocation to avoid outdated QA data sets and systems that are not synchronized with production.

Step #8: Claim Your Victory

Now that you have your test environment management situation under control and you’ve reduced contention between different projects it is time for you to lay claim to your success. Calculate the annual cost savings you’ve achieved either in terms of increased productivity or reduced spend on hardware. Make these figures visible to your management so that they can appreciate the increased productivity that is the natural outcome of your upgrades to test environment management.

Always take time to justify test environment management as these costs are often seen as unavoidable or hidden over the course of many years. Every CIO or CTO likes being surprised with new opportunities for cost savings and test environment management needs to be acknowledged as an important part of reducing waste within your department.

Step #9: Establish Regular Reviews

Polluted test environments are a natural by product of application development, and you’ll need to establish a weekly and monthly cadence of review sessions to make sure that all of your test environments are accounted for. Start a weekly meeting to assess test environments health and identify environments that may no longer be necessary. Then think about having a quarterly meeting to look at your test environment KPIs.

The best defence against Test Environment pollution is a good offence. Pay attention. Keep track of your policy and the exceptions to your policies. Communicate the rules of the game to your customers, and stay vigilant. Use Plutora to stay on top of test environment pollution.