The Regression Testing Solution for DevOps

Software testing is an essential part of any software development process to make sure everything works as expected. This concept is nothing new. However, what is the protocol for testing an application that is already in production?

New features are requested, bugs are reported — your team gets to work putting together new code and finding fixes. The next question is, “How thoroughly do we need to test this new code?” Sure, if we want to be really cautious we can test the entire application from ground up, but isn’t that level of extreme caution time-consuming, expensive and unnecessary? The answer might surprise you.

What is Regression Testing?

We’ll come back to the answer in a moment. First, let’s look at how regression testing is defined.

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software. Changes may include software enhancements, patches, configuration changes, etc. Source: Wikipedia

While this definition is a start, it still doesn’t tell us everything we need to know. Regression testing is typically split up into two types of tests. They are:

1. Tests that validate the functional use, and/or accurate processing of data, and

2. Tests that track performance figures.

Let’s look at an example. Imagine that our product is an online calculator. It has been up and running for several months now, and the calculations are 100% accurate. However, we have recently made some performance enhancements to the code: the calculated results should now display on the website twice as fast as before.

This means we will have two sets of tests to run. Of course, we would need to test the calculator’s performance to make sure the new code really is making results display twice as fast. But even more importantly, we need to test the accuracy of the calculator, to make sure that the new code has not caused unexpected calculation errors.  After all, to a customer, data accuracy is always more important than performance.

In this example, the testing of the performance is not considered regression testing as it is a new feature. Yes, it needs to be tested — but because it is being tested for the first time, that testing process is not considered regression testing. However, since the calculations should yield the same answers as before the code change, any testing performed that validates calculation accuracy is considered regression testing.

Why is Regression Testing Important?

Anytime a developer makes changes or enhancements to code, they are careful to not accidentally break or disrupt other functions of the application. However, application code is complex, and enterprise level application code can increase in complexity exponentially. Hundreds, if not thousands, of interdependencies can be in play. Even with the utmost care, problems still occur. It’s simply part of coding reality.

Knowing this, development teams perform regression testing to check and re-check the various components of the application following the release of any new code. This confirms (in theory) that their core functionality, performance and interdependencies still work correctly. They do this because the headlines are littered with big and small companies who have suffered major setbacks due to software glitches. Many, if not all, of these issues, could have been avoided with better testing. Comprehensive regression testing of an application with every release, before it goes live, is essential to avoid becoming the next headline.

Where does Regression Testing fit in the Development Lifecycle?

Whether you’re running an Agile or DevOps development environment, regression testing is always performed following integration testing and before user testing and deployment to production. While that part is pretty cut and dry, the bigger question is who is responsible for doing it. While it has traditionally been done by a dedicated tester, depending on your organization, it could be performed by a developer or even an automated test tool. There are even automated test tools that use artificial intelligence to perform regression testing. Typically, however, regression testing is performed by a test engineer who creates and manages the test cases, ensuring that all modules are effectively tested and working according to user specifications.

The key thing to remember is that regression testing is a key part of the release process — every bit as important as the development itself. It ensures that the application is working correctly, and that customers and business teams don’t experience problems that can get your organization in the news for the wrong reasons.

Effective Regression Testing in a Continuous Delivery DevOps Environment?

The stages of testing (shown in the above diagram) are identical regardless of whether your organization uses Agile or DevOps. The key difference between them arguably comes down to minor changes in the development lifecycle, and who is responsible for the various stages of testing.

With the Waterfall methodology the various testing stages were clear cut and well defined by different individual and team roles. Developers developed, testers tested and managers managed. However, with the continuous delivery and continuous testing model of Agile, those well-defined lines have become blurred. As companies try to improve development life-cycle efficiency, the responsibilities associated with each stage have shifted left.

In practice, this mean that developers were now taking on a significant amount of the testing to expedite issue resolution and code release. Then organizations began implementing DevOps with the goal of improving efficiency even more, which blurred the lines of finite roles even further and the responsibilities shifted even further to the left in the cycle. This meant that developers would take on more testing, and testers would take on more development. In some cases, even tasks that would traditionally have been the responsibility of operations would now be performed by testers and developers — interacting directly with the customer, identifying issues, and then quickly resolving them.

This is as far as we’ll go in this discussion comparing these two methodologies. However, even with these improved efficiencies and shortened timelines, the stages of testing haven’t changed and are just as important as ever. The only way to develop quality software as quickly and efficiently as possible is to provide efficient, effective regression testing.

Scheduling Environments for Regression Testing

Application testing in a large enterprise needs to be performed in a wide variety of environments that mimic the production environment as closely as possible. There are many ways to do this. One of the more popular methods is called Service Virtualization. In simple terms, this creates a virtual environment that simulates the production environment. This allows developers and testers to test their new code in an environment as close to live production as possible.

While that is the simplified explanation of Service Virtualization, the reality is that a global enterprise can have thousands of artifacts that make up a single virtual test environment. This means that to effectively mimic the production environment, everything needs to be considered — hardware, firmware, software, code versions, networks, and much more. When you factor in the needed hardware, software, resources, and licensing needed for every instance, regardless of production or test, it becomes financially unrealistic to expect a dedicated license for every test group. That means that these licenses have to be coordinated, scheduled and reserved for different developers and testers throughout the organization.

Without tools like Plutora Environments from Plutora, the coordination and scheduling of these environments is no small feat. Multiple development and test teams, thousands of different artifacts, and all of their various interdependencies —without the proper tools this can quickly become completely unmanageable.

Creating a Test Plan for Regression Testing

A quality Agile or DevOps program needs organization and structure. The same goes for the testing effort regardless of who is specifically assigned to run the tests. To make sure the testing is effective and thorough, a test plan needs to be created. A good test plan has multiple functions. They are:

Consistency – When performing regression testing, it’s important that the entire application is tested thoroughly.  One aspect of this is to ensure that every part of the application is tested. The other aspect is to make sure that the different parts of the application are tested effectively, using the same tests, test variables, and produce the same test results. Keeping and following a test plan like this will not only improve the speed in which the tests can be performed, but also ensure the accuracy of the tests, and thus the quality of the application.Show Testing Coverage – It’s almost impossible to show what parts of an application will be tested, or have been tested, without having a test plan to point to. It’s like a coloring book, showing an outline of what needs to be tested. As each test is performed, one section at a time is colored in. This makes it easy to see the status of the regression testing at any time, and if necessary, where to pick-up and continue the tests.Continuity – A good test plan should be completely transferable from one person to another. It should include every detail on how to exactly recreate the necessary test environment, set up the test scenario, perform the test, define data inputs, and identify what the results should be. This level of documentation will ensure continuity from one test series to another — and from one Test Engineer to another in case of reassignment.Speed – A good test plan significantly improves the speed and efficiency in which a regression test can be performed. This is because all the necessary tests and their respective details are laid out in a well thought out structure that is easy to read, duplicate and execute.Audit Trail – Enterprise organizations deal with high stakes. At some point, it will be necessary to have an audit trail to show what was done, by who, and when — whether to respond to a trivial query or to comply with a major investigation. A good test plan will show exactly that information.Accountability – Each test or test section should have space for the person who is performing the tests to record their name and date. This assigns an owner to each section who states “I have tested this application according to these recorded tests, and approve of it going into production.” When accountability is assigned in this way, that person takes a greater level of pride and responsibility in their work.

Maintaining an Effective Test Plan

Just as the code for an enterprise application is constantly changing and improving, effective regression testing on that application must also keep up with and adapt to the changes of the application. The regression test plans need to be maintained, not only to reflect new changes in the application code, but also to become iteratively more effective, thorough, and efficient. A Test Plan should be considered a living document.

Regression Testing Tools and Solutions

Automated TestThere are a variety of automated software testing tools to consider. They each have various claims of superiority. So, depending on your organization’s application, interface, network or methodology, there is more than likely an automated test solution waiting for you. For example, if you were needing to regularly perform regression tests on an online interface, Selenium, which specializes in web browser automation, would be a good solution to consider.Test MetricsPlutora automatically captures every detail and result from every test throughout the enterprise. Powerful reporting and analytics tools then organize and feed real-time data to reports and dashboards allowing testers, stakeholders and decision makers to know progress and results at any given moment. Additionally, Plutora has partnered with Tableau to give customers the most powerful analytics solution in the industry. This provides the ability to drill down into the data to further identify progress, coverage, areas of risk, KPIs, schedules, bottlenecks, and much more.Environment ManagementFor regression testing to be accurately performed on new application code, it needs to be tested in an environment that accurately mimics the production environment as closely as possible. For a global enterprise environment, this means mimicking thousands, if not millions of customers, and a dizzying network of servers, applications, firmware versions… a daunting task to say the least. These environments then need to be provisioned time and time again to meet the needs of the various development teams. It’s in this type of scenario that Plutora Environments shines. It effectively manages not only the environments but each of the thousands of different artifacts that comprise them. This allows you to quickly, reliably and repeatedly provision test environments that mimic every detail of the production environments, but without the risk.