Environment Management at Enterprise Scale
As a major utility company providing critical heating and electricity services to over 27,000,000 customers, service interruptions were not an option. Their enterprise production environment was hugely complex, with thousands of moving parts.
To comprehensively test new projects and applications before release, highly specific test environments were required. Every aspect of software, hardware, smart meters, firmware, code versions, customer volumes and traffic loads needed to be configured to the exact required specifications. There were over 1,900 components that needed to be factored into test environment configurations in order to accurately simulate the production environment.
As an additional challenge, they used production baseline environments right from the early stages of functional testing. Production-scale environments required significant pre-planning and configuration. They cannot be quickly spun-up quickly, yet they were required at every stage of testing. In 2017 alone, the environments delivery team processed over 700 requests for testing environments. Each request required precise configuration followed by a very detailed verification process involving cross-team coordination.
Fragmented and Opaque Scheduling
Each environment, along with its associated artifacts and configurations, was being tracked separately across four different business units and 20 different environment groups. Each business unit maintained their own set of siloed information which included itineraries, a series of spreadsheets and one or more Microsoft Access databases of metadata.
“It was a bit of a nightmare because we were using spreadsheets to manage the booking, locations, and scheduling of all these assets.”
Environments Delivery Assurance Manager
There was no clear consolidated picture of activities. Disconnected scheduling regularly caused conflicts and project delays. Accurate future planning based on consumption was impossible. Testing environments could not be properly audited, as there was nothing historically reliable to reference. Orchestration was carried out with in-person meetings, traditional email and manually updating spreadsheets — inevitably, details were lost along the way. To provision environments, they needed accurate views of current and target configurations, scheduling, booking requests and usage.
The Single Source of Truth
The organization turned to Plutora for test environment management. Once implemented, they consolidated the fragmented tracking from multiple spreadsheets and databases into a single location. Test environment management (TEM) managers in different teams were then able to draw from the same provisioned environments, artifacts and configuration settings. All the metadata, build details, code levels, firmware levels were fully integrated into Plutora allowing them to be easily and efficiently managed.
“The Calendar view of utilization was quite key, and fundamental to how we need to manage our environments.”
Technical Assurance Manager
The single calendar of all bookings provided value across all teams in the environment. While servers were tracked in the CMDB, Plutora became the single source of truth for an abstracted view of environments that included the grouping of components, configurations, software versions and connections as they would be available to development and test teams. Booking and change requests were tracked in a single location allowing environment managers to arbitrate conflicting requests in a single system. The single system provides a complete audit trail eliminating the varied set of forensic activities required to resolve misconfigurations.
Well-oiled Coordination and a Shift Left
Plutora enabled their product delivery team to shift-left by providing complete production baseline systems to the development teams at every stage of the development process. Development, automation and testing is performed against these environments shifting-left quality metrics to much closer to the development than ever before. As such, resolution times were reduced, product quality increased and product teams delivered more quickly and efficiently.
“With Plutora, we are able to set up specific environments, even for integration testing, which allows us to shift-Left even further, saving a lot of time and money in the release process.”
Environments Delivery Assurance Manager
What used to be nearly impossible to coordinate, became a well-oiled process managed by a single centralized solution and team. Environment requests were passed from ServiceNow and a self-service portal to Plutora where they were triaged. From there, the TEM team managed the requests and schedules with complete visibility across all resources and business groups. Longer running provisioning tasks for configuration, builds and test data were analyzed and automated including an integration to Jenkins as a job scheduler. Provisioning times were dramatically reduced and even included validation and verification steps ensuring a reliable environment every time. These benefits all added up enabling the development teams and operations teams to be fully integrated and coordinated like never before.
An Unexpected Expansion of Responsibility
With over 4 million smart meters installed in customers’ homes, the utility company needed a plan to ensure compatibility between devices already installed in-homes with the various IT delivery releases. A central part of the strategy involved comprehensive testing in their internal hardware lab. The lab had over 1200 unique assets with various firmware versions that must be validated against every release. Prior to Plutora, manual scheduling of these devices was problematic leading to conflicts and schedule delays.
The TEM team recognized that these physical hardware assets could be organized in much the same way as the software environment configurations. Hardware artifacts were then completely integrated into the Plutora TEM functionality enabling them to be allocated for testing across all business units. Scheduling conflicts were eliminated and remediated.
Visibility Empowered Future Planning
Plutora reporting and analytics gave the team a insights into consumption and utilization. The team utilized Plutora to have discussions about the historical, current and forecasted utilization of all systems. Test environment usage was consolidated leaving under-utilized systems decommissioned or repurposed. Systems could be blocked out for maintenance periods which ensured a higher level of reliability and resilience. The team was able to perform accurate capacity planning scaling systems to meet demand or to migrate to cloud-based services.
Analytics and what-if capabilities enabled the team to further drill into application and system impact matrices based upon upcoming changes. Future planning transformed from reactive delivery to strategic planning and service improvement with a focus on automation and deeper integration into DevOps tooling and testing activities.
“Every month one of the biggest questions to crop up would be ‘Do we have enough environments to cater for this change plan?’ We now get a single report with that information, it’s made life much simpler.”
Environments Delivery Assurance Manager
Collaboration and Efficiency
Plutora provided a record of accountability and activity that tracked the entire request, approval, scheduling, provisioning & change and decommissioning process. Instead of audit being a tedious record finding effort, environment management could hand auditors reporting into every aspect of every request and change.
Approvals and notifications were configured for stake holders eliminating the tedious process of constantly informing others on environment status. Management dashboards enabled a higher view of team member progress.
Having inbound requests arrive via self-service and ServiceNow into Plutora eliminated countless hours spent on the phone and email negotiating schedules and allocations. The implicit collaboration provided by Plutora created new levels of efficiency freeing the team to focus on more strategic planning and process improvement.