The Definitive Hiring Guide for Test Environment Managers
Mar 15, 2016
This is the second instalment in a series on hiring considerations within a modern enterprise software environment. For the first installment, click here.
Decades ago, when software projects were larger and more monolithic projects moved very slowly. Developers didn’t build micro-service architectures in weeks using modern software tools and processes such as Agile. They worked on projects with timelines measured in months or years.
Companies didn’t have a need to “manage” environments because there were only two – QA and Production - maybe three if you worked for a company with enough resources to run a staging environment. The days of the slow-moving software enterprise are long gone. The number of projects and dependencies has grown exponentially, and what was once one project has grown into hundreds of interconnected projects each with unique requirements for QA, staging, and production environments.
The software development process has accelerated and the demand for more and faster testing environments has increased. In this transition from one environment to hundreds, the industry has created a new job title – Test Environment Manager.
An Air Traffic Controller for Environments
Think of a test environment manager as an “air traffic controller” for environments and databases required to test and qualify software for release to production. This job is one focused primarily on tracking and scheduling, but it also involves integrating a number of conflicting inputs to support testing across multiple generations of interconnected systems. A test environment manager balances budgets with timelines and other constraints to give developers the systems they need to ensure that software works as designed in production.
As hardware and testing resources are finite a test environment manager is often responsible for adapting to aggressive timelines, forecasting hardware and database demand far in the future, and managing conflicts between groups with competing goals. When a team is ready to deliver software to production they often require test environments to be set up to exacting specifications. For this reason, you’ll want to hire a test environment manager who is a communicator and a project manager - someone with a focus on the details and who can adapt to the constantly shifting timelines that accompany software engineering.
Test Environment Manager Job Description
A Test Environment Manager has a primary task of preventing conflicts and collisions between multiple groups seeking access to limited resources. It is important to find the appropriate candidate with the right mix of technical expertise and people skills as the right hiring choice for test environment manager will result in fewer collisions, more efficient utilization of resources, and greater productivity. It is no exaggeration to say that hiring the right Test Environment Manager will help you get more value across your entire software organization.
Here’s what we would list if we were currently seeking the right candidate for a test environment manager position:
Seeking a qualified technical manager to coordinate test environments for multiple application development groups. Test environments for both isolated subsystems and end-to-end environments are configured using a mixture of cloud-based infrastructure and bare metal systems by a team responsible for release automation and configuration management. Test environments support a continuous release schedule across multiple groups. Platforms supported include Java, .NET, and a number of dynamic languages all running on applications servers integrated with several relational databases running DB2, Oracle, and SQL Server. Test environment managers will interact with project managers, operations professionals, and senior technical engineering managers to sequence projects for testing on a limited number of testing environments.
Candidates must have demonstrated the ability to understand modern approaches to deployment automation on both cloud-based infrastructure and bare-metal servers and work closely with release managers and release engineers to track the availability of limited testing resources. Technical tasks such as deployment and database administration will be a product of close collaboration with engineering resources and database specialists, but candidates should have an overall grasp of the work-effort involved to setup and configure operating systems, networks, databases, and application servers involved in the creation of end-to-end testing environments. Candidates will be managing the coordination of efforts to configure environments and tracking the status of environment configuration activities.
Candidates must have a demonstrated track record of organizing and tracking the progress of multiple projects across the software development lifecycle. As this job involves interaction with engineering managers and project managers ideal candidates for the test environment manager position will have experience communicating with management on topics including project timelines, contingency plans, environment budgets, and resource allocation. The ability to use project planning and forecasting tools is a requirement.
You are looking for a candidate who understands the technical work required to stand-up test environments, but who is ready to delegate this work to associated release engineering groups. They shouldn’t be eager to jump in and stand up test environments themselves, but they need to understand how long it should take to configure a database or setup and configure a number of VMs to facilitate testing.
Roles and Responsibilities
Develop process to track test environments required for QA
Create short-term plans to deliver environments to support sprint-based development
Coordinate activities of release engineers to automate the setup and configuration of test environments
Monitor and guarantee uptime of test environments
Provide ongoing support for test environments
Communicate test environment availability to project management and quality assurance
Develop KPIs to track efficacy of test environment delivery efforts
Resolve conflicts between teams competing for limited testing resources
Move test environment activities toward a goal of 100% automation
Provide estimates for onboarding of new projects to test environments
Design systems to support continuous delivery and adapt test environment management to support on-demand, self-service automation
Forecast environment requirements based on anticipated future demand from multiple application development teams
What Skills to Look for When Hiring a Test Environment Manager?
Ability to communicate with stakeholders: project management, QA, and engineering management
Focus on scheduling accuracy with the ability to keep a constantly shifting test environment calendar up to date
Ability to estimate work effort required to bring test environments to readiness
Focus on organization and communication over technology – technology is important for a test environment manager, but communication is the critical skill.
Who is the Ideal Candidate?
Test environment managers sit at the intersection of several groups – DBAs, application developers, network engineers, and management. Your ideal candidate will be someone familiar with all of these groups, and the most ideal candidate in a large enterprise will be an internal hire – someone who has already been communicating with internal customers and who understands the existing dynamics between teams.
If it isn’t possible to hire internally a good place to start looking is project management. Project managers are often forced to juggle competing inputs and that’s the primary skill of a test environment manager. The ability to understand both the budget and time constraints that drive the decisions to prioritize one project over another is the skill that will allow a test environment manager to excel. A project manager who has been working with technology long enough will have the requisite understanding of technology. They will understand the challenges faced by QA, Network Engineering, system administrators, and DBAs, and they have had years of experience managing complex and shifting timelines. If you can find a project manager with a focus on technology who is interested in thinking about projects at the portfolio-level, this is your ideal candidate for the test environment manager.
Another good place to look for test environment managers is Quality Engineering. These are the more technical resources involved with testing who have experience standing up and tearing down environments used for QA. Quality engineers often have direct experience working with individual environments and they have an understanding of both the complexities involved with environment management as well as the time constraints of QA. As test environment managers will be interacting with QA these candidates come with a built-in advantage: they understand the customer.
Who Not to Hire?
Technical familiarity is key, but you don’t want to hire a purely technical resource for this role. Don’t hire a release engineer for this role, as a test environment manager should be more focused on delegation and communication than on technical tasks like configuration and administration. Find someone a bit more technical than your project manager and far less technical than your release engineers focused on automation scripts.
If you hire someone not technical enough they won’t be able to understand when teams are giving them unrealistic estimates (either too short or too long.) If you hire someone too technical they will be stuck in the details of environment automation while teams of developers competing for resources feel neglected and miscommunicated with. You want someone just technical enough to understand the work, but not too technical to get distracted.
Remember, a test environment manager is more about sequencing and scheduling than anything else. You are looking for someone who values a schedule and who is able to juggle projects across limited resources to get the most efficiency out of your hardware resources.