What is Test Environment Pollution?

Test environment pollution and test environment churn are two of the biggest factors contributing to delay and inefficiency in test environment management. Test environment churn is a measure of how quickly test environments are changing and the stability of your test environments. Test environment pollution is a measure of how active, up-to-date, and relevant your test environments are across multiple projects. This post focuses on the different kinds of environment pollution and briefly discusses some of the contributing factors for each.

Environment Pollution: Unused Environments

This is the most common form of environment pollution: environments that sit for weeks, months, or years without being utilized. If you work at a large organization that lacks the discipline to keep track of test environments you’ll encounter test environments that sit idle because no one bothered to inform a test environment manager that they were no longer being used.

Unused environments are very common in organizations that give development teams access to self-service provisioning tools. Without strong oversight teams frequently over-provision test environments and move on without leaving a note. This is why every test manager should invest in systems to track release and usage activities on test environments. Idle environments consume resources that could be used for other purposes.

Environment Pollution: Misnamed Environments

Modern software architecture is so dynamic and agile development processes can move so quickly that teams can be presented with the challenge of having to deploy multiple branches on a limited number of environments. An example of this is using a staging environment for QA or using a production environment for staging. Some organizations sidestep this problem by using generic names for environments like S1, S2, S3, and so forth, while other organizations name environments after the type of activity they support such as QA01, QA02, Stage01, and Production01.

Both approaches have advantages and disadvantages, but common anti-pattern is to use named environments for the wrong kind of testing activity. Any time anyone tells you that a production deployment needs to be “staged” to production and that the staging environment supports QA you’ll understand how confusing this practice can be over time. If production is staging and staging is QA what happens on QA? A different kind of QA? This creates environment pollution because no one is ever certain if an environment’s name reflects its purpose.

Environment Pollution: Mixed Environment Testing

If you have multiple environment types such as QA, Staging, and Production but you only maintain a single production database to which they all point to you are practicing mixed environment testing. It is a dangerous practice mix QA with production and when you do this you run the risk of losing production data during a QA test run or inadvertently configuring a production system to point to a QA version of a service.

Mixed environment testing happens where there is little or no segmentation between production networks and testing networks, and in multi-level, microservice architectures there is always an option to just point a QA system to production-level backend services. Mixed environment testing often limits the testing activities you can perform because whether you are mixing staging data with QA data or production data with QA data you need to always be careful not to affect production.

Environment Pollution: Unsynchronized Environments

When a test environment is so far out of sync from production in terms of configuration it isn’t a test environment at all. Test environments should match the configuration of production systems. Take a web site as an example. If your production systems are configured to use compression to serve web content, your QA and staging environments should be similarly configured. If your application is tuned to use a certain approach to connecting to a database in production you should use the same approach in QA and staging.

When your QA and Staging systems are not configured to match your production systems you are not creating an environment that supports QA. When you test on unsynchronized environments you will be constantly surprised by bugs that appear in production only because no one was able to test a specific approach to configuration.

Environment Pollution: Test Environments with Production Data

In certain industries testing with real production customer data runs counter to several regulations, and, in some cases, the law. In less-regulated industries testing with production data is very common. The idea behind building a test environment with production data is that your QA process should be able to test against a carbon-copy of the data that will be encountered in production. While it makes it easier to write accurate tests there are numerous downsides to running QA with production data.

If you follow this practice you run the risk of running a test that may impact real customers. For example, you could run a test of an email notification process that sends real emails to end-users because you haven’t properly masked or scrubbed the data being used. You also run the risk of exposing sensitive customer data unnecessarily. Again, in certain industries like healthcare and banking this isn’t just a bad idea, it is illegal. Avoid testing with production data.

Environment Pollution: Unpatched, Insecure Testing Environments

If your organization worries about security patches and tracking vulnerabilities in production you should have processes in place to run these patches on QA and Staging, and you should have an intentional approach to securing your QA and Staging environments in the same way that you protect production.

This is important because you want to synchronize environment configuration to create more “accurate” test environments, but it is also important because QA and staging environments are often an easy way for hackers to infiltrate an organization. Companies such as Google understand that developer laptops are a popular target for hackers because they are often one step away from loosely protected source code management, QA, and staging systems.

Environment Pollution: Out-of-date Testing Data Sets

At the other end of the spectrum from organizations that test against production data are organizations that configure test environments with test data that is years out-of-date. Testing against ancient data is testing nothing at all. Production systems serving real customers generate patterns that need to be tested by test cases on non-production systems. This is especially true for large, public-facing web sites: users sign-up with uncommon names containing special characters and real users often configure applications in unanticipated ways.

To create test environments that support accurate testing you should be running regular automated processes to copy and mask data from production. If you don’t test with data sets similar to production you run the risk of creating more unused environments. With ancient data no one will trust your test environments, and tests run on these systems will be so unreliable no one will use these test environments.

Use Plutora to Track and Reduce Test Environment Pollution

Plutora gives you the ability to track test environments across multiple projects and departments. You can keep track of environment churn and pollution and use Plutora as a way to identify unused environments or environment contention that often leads to dangerous, mixed environment testing practices.

Plutora allows you to model the effort required to refresh testing data and gives you a valuable tool to govern and communicate across hundreds of test environment management customers. It is a tool to facilitate a more rational approach to environment management that encourages greater efficiency and less waste.

Don’t accept test environment pollution. Clean it up with Plutora.