Steps to Maturing Test Automation within a Deployment Pipeline

It is also true that by trying to automate too much (or the wrong things) too fast, you can set yourself back a long way.

With that said, automation is a tool that all companies should prioritize when operating in an agile development environment. How you go about setting up your automation practice will determine just how many of the benefits you can realize.

Remember, you have to crawl before you can walk and walk before you can sprint. To help move you along, I have outlined some examples that detail the steps you should take along the SDLC to help your deployment pipeline evolve from beginner to intermediate to expert.

Beginner Stage

In this beginner stage, it is all about creating quick wins for your team. You will want your development team to begin by checking in their tests and immediately provide pass-fail feedback. Once unit tests are passing, you can get started with a very small set of automated smoke tests.

Initially it may be easier to run these tests at night, but you will eventually want these running as part of the build process so they are automatically triggered upon a successful build. Once they are passing with consistency, you can incrementally add more smoke tests to your automation suite.

Adding one small test at a time will allow you to gain confidence quickly in your automation, as it establishes the building blocks for your automation practice. Not only does this build confidence, but it saves time in the long-run from having go back and fix a larger set of tests.

Intermediate Stage

Now that you have built some confidence in your initial set of smoke sets, you can begin to attack additional layers of the application. For example, if you have an application that allows for you to test both integrated and UI tests, this would be a great opportunity to fan out your tests and run them in parallel with each other. The more you can cover of the functional testing pyramid, the more confidence you can have going into your regression tests.

Your regression tests are where you can really start running multiple layers in parallel to decrease the time to get results and feedback on the quality of the application. Like your automated smoke tests, your automated regression tests will grow bit by bit until you’ve reached a comfortable level of test coverage and can begin optimizing.

To complement your automated regression tests, your manual team should prioritize any manual tests as well as continuous manual exploratory tests. The greater percentage of regression tests you can automate, the more time you can spend on exploratory testing, increasing the coverage of the edge cases where interesting bugs usually hide.

The most important competency of this intermediate SDLC stage is that you establish a test case management system and reporting structure. It’s key to make sure that all of your results are going to a single place to be seen from a single view. By doing that, you have a consistent view of any failures and can comfortably decide whether or not to deploy.

Expert Stage

The biggest difference between the intermediate and expert stages is the implementation of quality gates. This aligns with your need for a test case management system, as quality gates will allow for you to reject any builds before deploying or moving on to further testing.

Again, your goal here is to provide rapid, early feedback to your development team so they can perform timely root cause analysis. In doing so, you ensure critical bugs do not make it further along into development, eliminating more costly and timely fixes down the road.

The key here is to do just enough testing at each phase before you expand and fan out your testing. There is a lot that needs to be covered, so find a comfortable point before adding in new test environments.

Lastly, you want to build confidence at each quality gate before running more time-consuming, costly automation. Oftentimes when you run those time-consuming tests, it can end up being a bottleneck and slowing you down. Doing so further down the road gives you more confidence once you’ve gotten there.

Wrapping Up

Taking your deployment pipeline from beginner to expert does not happen overnight. It requires a commitment to the process and patience for moving ahead incrementally. That commitment includes an investment in technology, an established set of processes, and an ability to track and report on your progress.

Though this all may seem like a daunting task (and it is), many of the struggles companies face with automation can be mitigated with crowdtesting resources. With the resources you need, your deployment pipeline will look like an expert’s in no time.