Experts Discuss How to Improve Testing Efficiency

In our age of fast release cycles, software development organizations must conduct their work quickly and efficiently.

The same goes for software testers, who are often short on time to complete their tasks — even if digital quality is perhaps the most important aspect of a release. Software testing efficiency means making the most of the available resources to put a high quality product in the hands of customers, one that they will enjoy using. Anything short of that outcome risks lost revenue and project failure.

In an economic period with tight budgets, what are some ways to promote more efficient software testing? In these snippets from the Ready, Test, Go. podcast, expert guests comment on some ways to be more shrewd and resourceful in how testers can approach their work.

Quality assurance expert Amy Reichert provided some background for how to prioritize tests, multi-task, and keep calm in high-pressure situations.

(quotes have been edited for brevity)

Amy Reichert, Freelance QA SME/Test Engineer

I think it depends on the experience of your team. So if you have an experienced testing team, or at least maybe not so much experience, but familiarity with the application, both the back end and the front end — the more experience or understanding of the application the tester has [enables them to] go off script and basically use different techniques. So instead of testing script by script by script and prioritizing those scripts, you can prioritize the functional areas you want to test.

What I really like to do is use exploratory testing in conjunction with automated smoke tests or automated tests, regression tests, if those exist. So when the developers are testing — I mean, we usually start with starting the automation, but we don’t wait for it to just end. We just go ahead and jump in with the exploratory testing because automated tests will fail sometimes for no particular reason, and it sucks down a lot of time to figure out why they failed, if it’s really for a script reason or an actual defect. So what we’ll do is we’ll kick them off, and then we will go ahead and start our exploratory testing. And that kind of helps you save time because then I can take the exploratory tests and use them to cover a lot more test coverage, if that makes sense. I can cover the UI, the back and the front end, all of it without interrupting my flow.

I think you just keep going. It takes some perseverance, and staying calm and not getting sucked into the chaos — and just steady on testing. Use your exploratory tests, have someone check on the automation. We just keep soldiering on, I guess you could call it.

How much value are you getting out of each type of testing? Many organizations have no clue. Matt Heusser, Managing Director of Excelon Development, recommends evaluating all of your testing activities to determine the value they provide.

Matt Heusser, Managing Director of Excelon Development

Where do you start? Let’s look at all of our testing activities. We do usability, accessibility, performance, load, unit, system, integration, outsourced, crowdsourced. How do you possibly compare all of them? They go in different buckets. They have different managers. They have different budgets. Where do you start?

I would start with the bugs, right? So to go into our bug tracking system or whatever it is. Maybe you’re a high functioning Agile team, and you fix them the day you find them. Track them for two weeks, man. It’s two weeks. Get a list that says, ‘Where was the defect created? Where was it found? Is that right?’ And you can sort that spreadsheet by [those criteria]. Often, this problem should have been found in design. Any time we have a tester arguing with a developer about what it should do, it’s a design or requirements problem because we’re clear on what it should do from the beginning. We get that list and then we can start to look at the priorities and effectiveness of our test efforts. So we can say, this tough effort is not valuable. None of the defects that are emerging are the kind that this thing should be checking.

If you look at how many organizations do testing, what we do is, we say, we’re doing stories or micro features, whatever you want to call them. Then, for every story or micro feature, we assign someone to do the testing, and that person does whatever they do. And then, it’s tested. And then, maybe, at the end, we have some regression process that goes through and makes sure that they change this. It didn’t break something over there, unintended. It’s all tested about the same. All the testing is about this wide. I’m going to talk about some numbers, as if it were possible for us to model, that it’s very difficult to do. What these numbers are going to be approximations. It’s going to feel like you’re loaded on the back of a napkin in a bar over a beer — you’ll be embarrassed, but it’s going to be better than nothing.

So with a risk-managed approach, you look at the impact if the problem happened, percentage chance that’s going to be introduced into the world. Multiply those two numbers with all of your senses of risks, and then you sort. What will happen is, the most likely problems with the highest impact will go to the top. Then, you can invest your scarce resources and go on working down that list. So at least, the bottom might get a lot less attention than the things at the top. What that would mean for testing would be, let’s look at these stories. In e-commerce, that’s usually half the purchase. So, search [functionality] has got to a work or they can’t find the product. Add to cart [functionality] has got to work. Checkout has [to work]. If these things don’t work…you can’t buy product. Every minute, you lose.

But if we just always do what we always did, everything gets about this much testing. And then, we’re missing the opportunity to reprioritize to catch the defects that have a disproportionate impact on our customer base.

Organizational methodologies such as Agile and DevOps can help break down traditional silos to shorten product development life cycles. However, these approaches carry some formalities with them that can be misunderstood or misused. Jeff Payne, CEO of Coveros, discussed one such Agile ceremony intended to improve digital quality that actually inhibits it.

Jeff Payne, CEO of Coveros

Organizations are trying to fix the problem that they don’t have time to finish their testing by making the sprints longer. Well, you’re kind of solving the wrong problem when you’re doing that. It’s usually not the duration of the sprint. It’s how you’re working in that sprint, right?

Or, you’ll hear organizations who every so often will have a hardening sprint, they’ll call it. They only call it that because they know you’re not supposed to have testing sprints. So, they call it “hardening” instead. But it’s just basically catch-up on all the testing we didn’t get done, or fix all the bugs we still have.

And, again, solving the wrong problem, right? We need to figure out how to work better in our sprints to fix those kinds of things. Those are some red flags that we see a lot with people that we talk to and that are struggling with Agile and Agile testing.

For me, the thing that has been the most successful is some form of pairing between devs and testers. I wrote an article on dev-test pairing and gave some different approaches that you could use to get developers and testers working together. Whether it’s one-on-one pairing with a dev and a test, whether it’s using kind of the BDD Three Amigos [approach], where you’ve got the business, dev and test all working together on a story, or you’re mobbing where you have everybody working to build and test story by story. Whatever you’re doing to get people to work together every day in some model is going to help that a lot, right?

Make a contest of it too. I’ve seen success with creating what we call a pairing board, where you take your team and you say, all right, developers are on this axis, my testers and other roles are on this axis, and we’re going to set a goal. Depending on the size of your team, every sprint or every quarterly increment or whatever the time frame is, we’re going to figure out how everybody works with everybody else at least once on a story. And we’re going to fill the chart in, and we’ll do it at the end of every — either in your stand ups if it’s just a sprint activity, or at your retros or your sprint demos. Then, if we fill in the whole chart at the end of the quarter, we’re going to have a party, or something. Or you’re going to get a prize, or whatever it is, right? Gamify it, make it fun, make it something to track. I’ve seen success with that because then it’s a fun gaming activity with maybe some reward at the end of it, instead of a, “thou shall all work together,” kind of a mantra from above.

Organizations must carry an adaptive mindset to uncover new bugs in the future. Work for the sake of doing work isn’t enough to promote software quality. James Mortlock, Lead UAT Test Manager at Vodafone, discussed this idea, as well as why efficiency benefits everyone up and down the business.

James Mortlock, Lead UAT Test Manager at Vodafone

Be really, really efficient with it. It’s the pesticide paradox, right, if you keep doing the same thing over and over and over again, and the bugs are going to get resistant, if you’re talking about real life. But you’re not going to find anything. All the bugs will be where you’re not testing. So if you focus on– that’s where you’ve built the quality really, really high– if you focus on that all the time, the bugs will be outside of this area. So stop testing there, start testing somewhere else. And if you start to see a re-emergence of issues and bugs, then that’s where you have to be flexible and re-implement new strategies and increase quality assurance in that area.

If you can make efficiencies in costs, you’re going to probably get stuff signed off. But with that comes– like say you’ve got a test environment that’s just– like we’ve got a place called Spaghetti Junction, and it’s just a overlap and cross of roads and bridges and whatnot. If you’ve got an environment that’s like that, it’s going to be a nightmare being able to keep it in sync and your cycle time instead of being like a day will be like 10 days because the intricacies are so huge. You go through and rip that out and make it streamlined and efficient, you’re going to save the business days and days and days. But also, your developers and your QAs are going to be so much happier. Because being able to just test something or just deploy something is going to be so much easier. So you save the business loads of money, then all of your employees within development are way happier, therefore they’re more productive, therefore you make more money. So it’s like the infinite reward.

For more digital quality insights, subscribe to the Ready, Test, Go. podcast on your favorite podcast app, or watch episodes on our YouTube channel.