Why Now is the Time for Test Suite Maintenance

It is often embarrassing for a QA manager to admit when it is time for test suite maintenance.

We’ve all been there. During the lifetime of an application we’ve written test cases for version 1.0 features, creating our first regression test suite. But, as the application matures, more features are added, GUI layouts get changed, and new technologies are incorporated to build the latest and greatest application update: version 2.0. Later, testers write test cases for new features as part of project delivery, which they test thoroughly and eventually move into the regression suite as we plan for the exciting version 3.0.

Here is where questions around test suite maintenance come into play:

  • What about those version 1.0 test cases? Are those older test cases still valid?

  • While building the latest version, did we make any changes to the application that affect the steps or expected outcome of those test cases as written? Probably.

  • Did we update those test cases to reflect the new design? Probably not.

  • Will testers know that we changed that feature and not report those failed tests as bugs? Sure, if testers are a constant in the life cycle of the application, they might. But chances are that some testers won’t be around the next time the application must be regressed.

Any number of factors can put you in the position to hand the test suite over to a new group of testers that have never seen your application, including personnel changes, stakeholder strategy changes and corporate mergers. In these instances, an out-of-date test suite will cost time and money in the form of bug reports, which will ultimately end up being filed as Working as Designed or Deprecated during bug triage.

We get it. The shiny new features are exciting, and more pressing. There are deadlines to meet, and there is very little time to write test cases for the new version, let alone update the existing test suite. “We’ll have time to update after we finish this sprint,” some managers tell the team. However, the next sprint is already in the planning stage, with even more exciting new features. That time never materializes.

During my time with Applause, I’ve helped many companies rebuild, update and maintain test suites. I’ve seen it all, from outdated feature tests to test cases with only a title and a link to a ticket in Jira — which I didn’t have access to! I’ve seen regression suites contain multiple examples of the exact same test case, and I don’t how many times I’ve run into the unhelpful “TBD” in the steps. All of these test inefficiencies waste valuable testing time by causing a tester to come back for clarification; or, because the tester followed the steps to the letter, they logged a bug that gets ignored later on. Any of these scenarios should trigger test case maintenance, but the bug is often just closed with every intention to update the test case later. Later never arrives.

Test case maintenance is an area where Applause can make an impact on suites in a short amount of time. Our teams approach test cases with fresh eyes and can quickly identify all these issues, documenting the outdated and identifying missing or inaccurate information. As we learn the current state of an application, Applause works with you on test suite maintenance — and, in most cases, we can even make it more efficient.

One client of mine handed my team a test suite housed in TestRail that contained about 1,600 test cases. By the time we finished our review, we reduced the overall test case count to roughly 600 test cases, which ended up covering more functionality than the original 1,600 count. Applause achieved this by identifying and removing deprecated cases, updating the test steps of stale but still valid cases, and combining test cases. When it comes to test case combination, the only difference was the platform on which the test case was executed. From there, we applied a labeling structure to each, which enabled faster test plan creation.

Your time is valuable. The longer you put off test suite maintenance, the more time you will lose to the slow bleed caused by inaccurate test cases. Let Applause help you get your time back.


Three Best Practices For Test Case Management

Part of a QA organization’s success depends on test case management. Learn the challenges that software dev orgs face in managing TCM and see three best practices.

Read 'Three Best Practices For Test Case Management' Now
Want to see more like this?
John Kotzian
Test Architect
Reading time: 4 min

Anatomy of a Streaming Media Company Pilot

Gain insight into what a pilot engagement looks like with a global media company to assess their digital accessibility maturity

7 Habits of Highly Successful Mobile App Launches

These criteria go a long way toward winning with mobile users.

How to Achieve the SAFe Core Value of Built-In Quality

Learn how Audi made built-in quality in the Scaled Agile Framework a success

Why Live-Event Testing Is an Imperative for Media Companies

Streaming audiences are demanding more live content — delivered without defects

Four Ways Crowdtesting Drives Operational Efficiency and Cost-Effective Development

Read about the ways crowdtesting can help firms produce high-quality digital experiences while keeping costs in check

How to Onboard a Crowdtesting Partner Effectively

Guidance on getting started with a crowdtesting vendor to improve your organization’s quality, speed, and agility.