How to Build a Test Strategy Around Automation

When should you use manual functional testing, and when should you rely on automated testing?

As your product matures, so does your ability to move to more automated testing. In a recent webinar, Dean Vittum, Applause’s Manager of QA, and Cathy Huang, Applause Product Director, reviewed what automation should look like at each maturity stage, and how to develop an automation strategy at each stage. In other words, how testing strategy differs for a brand-new product vs. a growth product vs. a mature product.

Create a test strategy for a new product

A new product is one which is still in the early stages of maturity. There are few areas you have tested regularly, with new additions and adjustments still coming down the line. In particular, the UI is often not set in stone at this phase.

The instability of an immature product is one reason why you should not do a ton of automated testing at this stage. You need to give your product the chance to mature before introducing automated testing.

“Historically, I like to see things initially [manually] tested over a certain period of time so that you’re not wasting your time scripting something and then finding out a couple days later or a couple weeks later, or a build or two later that things have changed,” Vittum said. “In that case, you would have to go back and re-script everything.”

So if you aren’t utilizing automation for new products, what types of testing do you want to focus on?

“The test strategy that we want to cover here is that you start with exploratory testing,” Huang said. “And the reason why is because you might not have a full set of requirements that you’re working from, but you can walk through the product like a user.”

Update your testing strategy for the growth stage

As your product spends more time in the market, it enters the growth maturity stage. In this stage, your product has several areas which you have tested thoroughly and are deeply familiar with. Your product stays in this stage for some time as you iron out the details of the UI and add new features.

The growth stage presents an opportunity to expand the role of automation. Huang’s suggestion is to start with what you know. Automate those structured tests you developed in the previous maturity stage which you know you can rely on.

“Take the test cases that have shown the test of time,” Huang said, adding, ”You’re going to get more value from it, because they’re touched quite a bit. Start automating there before going to some of the edge cases.”

However, don’t forget about manual exploratory testing.

“As you mature, exploratory testing is still an important part of this (test strategy) process,” Vittum said.

As you find bugs through exploratory testing, you can take those steps for reproduction and turn it into a test case, then add it to your regression suite. “If you start to have any additional test cases you add those to your full suite, or into your regression test suite.” Vittum said.

You need to give your product the chance to mature before introducing automated testing.

Finalize automation for a mature product

Lastly, we look at a product in its final stages of maturity. This is a product which has been on the market for years, and you’ve likely put the big hurdles behind you.

“But, at the same time you may have fallen behind in your (testing) practices,” Huang says of the maturity stage. “Now is a good time to step back and see where you are missing some of your core testing strategies.” Huang suggests starting with the basics. Check on unit tests as they are likely outdated at this stage.

Once you feel confident in the balance you reach with your test suite, it is time to really ramp up automation. For a mature product, Vittum says you’re “doing really well” if you automate 40% to 50% of the testing.

How should you get started? Vittum advises, “You may want to look at what you have in place, how time consuming it might be for your manual testing, and what kind of time you could save if you were to implement an automation tool.”

Vittum adds that the team here at Applause practices what we preach by “drinking our own champagne.” While “dogfooding” isn’t always a great standalone QA practice, it can be effective as one piece of a testing strategy. When Applause is testing new builds, the team runs manual and exploratory test cycles with both our internal QA teams and the Applause Community of vetted QA testers.

“Once we have seen a historical series of successful test cycles completed, we then turn our attention and focus on the highest vulnerable areas of the application to convert into automation scripts - eliminating those long regression test cycles before releasing,” Vittum said.

To get more details on Cathy and Dean’s approach to building a testing strategy around automation, watch their webinar, Bringing Balance to Manual and Automated Testing.