How to Prepare for Test Environment Emergencies

Environment “shortages” always start with a test environment emergency – some unforeseen need for a new environment. Maybe a high-profile project has both an urgent production bug to fix and an unrealistic deadline to meet at the same time. This project’s QA and staging environments can’t be disturbed to fix production without ruining a schedule and someone approaches environment management with a request: “Can we get another environment set up, quickly?”

As an environment manager it is your job to plan ahead and enforce policy, but there are just some battles you are not going to win. This is one of them. While you can tell the project that no environments are available you’ll end up getting blamed when either the production bug isn’t fixed or the project can’t meet a critical deadline.

How you prepare for emergency environment requests makes the difference between your success and failure as an environment manager. Here are some steps you can take to prepare for emergency environment requests:

Survey your biggest customers and plan for the unexpected. The larger the project the more unpredictable your environment demand will be. While some projects are very straightforward every organization has a few “mega-projects” projects like a central web site that have a dynamic release schedule supported by hundreds or thousands of developers. Make sure to reserve capacity for unexpected scheduling changes or bugs for these larger, more-dynamic projects.Set aside some hardware and resources for the unexpected. Model your application’s needs and set aside enough excess capacity to deal with unexpected situations. If you are developing a web site that interacts with services make sure that you could spin up a separate environment for all of your system components and make sure that you never reach 100% allocation of existing hardware or cloud-based resources.Plan for two test environment emergencies. Emergency environment requests are often made in response to a critical production bug. “We can’t disrupt ongoing development, and we need a new environment ASAP.” When problems start happening in complex systems they tend to happen in clusters and you need to be ready to handle more than one unanticipated emergency at once.Use cloud-based resources as your emergency “chute.” Today’s large enterprise is a hybrid of in-house resources and systems running on a public-cloud. If you have access to a public-cloud such as AWS or Azure take the time to deploy staging and QA environments to a public cloud. This will make it easier to add additional capacity for an application as VM resources can be created quickly.Complex projects often require an emergency environment on standby. If you have a web site that delivers to production in weekly or biweekly delivery cycles you may already have an emergency environment on standby. You need to guide your customers through a cost-benefit analysis of maintaining a dedicated emergency fix environment. If a project wants to maintain an aggressive release cadence while maintaining the ability to respond to production bugs quickly you’ll need to budget for an emergency standby environment. Emergency standby environments will often seem like a waste of resources, but they are a necessary insurance policy that will make it possible to handle and react to an emergency without disrupting ongoing development.Don’t advertise your excess environment budget. If you’ve planned ahead and you have enough hardware to support a few emergency test environments make sure you don’t openly advertise this. Large organizations are always hungry for more resources and as soon as you advertise that hardware is available projects will request it. Keep your emergency supplies available for a real emergency or else they will become some project’s third QA environment (that will go unused for several months.)Use Plutora to plan out your Environment Management needs. Our system allows you to model the environment requirements of every application team independently. Once you’ve modeled these requirements at the project level you can take a step back and assess your entire organization’s environment requirements. For many of our customers it is the first time they are able to accurately predict what it will take to support hundreds of projects across several departments. With Plutora you can create more accurate environment forecasts and you can predict which projects are going to have conflicting environment requirements. With this insight you can avoid test environment emergencies.