The Why of Software Testing

Transcript

DAVID CARTY: At the end of the day, some people prefer to relax with a glass of wine and their favorite TV show, if they have access to the remote control, but other people prefer to keep their hands busy and their minds active. That’s what led James Mortlock to leathercrafting.

JAMES MORTLOCK: I stumbled upon a specific YouTuber that did just no talking, just making leathercraft stuff, and it was beautifully shot and the audio was amazing. This was before everyone was playing ASMR across everything. It was beautiful, and I thought, how hard could it be? I tried it, very hard, and I just got less and less bad at it, effectively. There’s like the kind of creativity side of my, I guess of my brain, that I don’t necessarily always get to explore with every single part of my job. So it’s good to be able to flex that sort of creativity side of the brain.

CARTY: There’s something meditative in the hobby for James. Who knew making a wallet could bring you one step closer to enlightenment?

MORTLOCK: I’m not a big fan of just sitting, watching TV, and that sort of stuff. I need to be kind of being productive and creative and doing something. So this was good because I could sit on the sofa and do it with other people around, and it’s very meditative. I do sort of mental health neurodiversity talks and stuff like that, and talking about meditation, about focusing the non-complicated side of your brain and turning the complicated one off. That’s like I’ve got to pay this bill and these need washing and the kids need picking up, all that sort of stuff, that you turn that side off and just let the left hand, right hand, left hand, right hand, take over and you just feel yourself unwind a little bit. So, yeah, it’s meditative and keeps me busy.

CARTY: And the rewarding part of leathercrafting is that it yields a tangible output, something you can hold in your hand or give to a loved one, not to mention that it’s opened up James to a new community of friends with a shared interest.

MORTLOCK: It started out just making like really basic card holders, so like three pockets, one either side and one in the middle type thing, practicing stitching, and yeah, it evolved into things like mugs, wallets, and backpacks and stuff like that. So, yeah, just wherever my brain sparked off to, do a sketch, and then try and turn it into a reality, which is a lot harder than just writing it down. It’s easier said than done, comes to mind. I, myself, am neurodiverse, so that whole being able to have something to focus on that has a tangible output, that if you need something quick, you can make something quick, and gets you that really– yeah, you can feel literally and figuratively and emotionally. You can feel that sense of achievement. When I start talking about something that I know bits and pieces on or I’m passionate about, there’s no stopping me, like going and meeting like minded people and people coming to my stall and asking me about stuff. Yeah, I think that’s the real achievement there, is being able to connect with other people and share a little dark corner of the world or illuminate a dark corner of the world that you didn’t really know existed.

CARTY: This is the Ready, Test, Go. podcast, brought to you by Applause. I’m David Carty.

Today’s guest is a leathercrafter and Lead UAT Test Manager at Vodafone, James Mortlock. Beyond his testing background, James has experience with automation. In fact, he constructed the first UAT automation of voice assistants via Raspberry Pi. He was also a product owner for analytics implementation, and he was even a Scrum Master. So, you can believe him when he says, the purpose of software testing is often poorly understood across the organization and, sometimes, even by testers themselves. It’s easy to fall into a pattern of behavior and never question why you are performing a certain task and whether it delivers any real value. So, how should we reframe how we look at this critical task of software testing to ultimately deliver more value with the approach? Well, that’s for James to know and for us to find out.

James, I love the premise of this episode, The Why of Software Testing. So let’s start by answering that question, why do we test software and what should be the stated goal of testing software?

MORTLOCK: So, yeah, interesting question, and it can be, I guess, as deep and as shallow as you want it to be. But I guess the why came from probably came from my personality a little bit because it’s just this constant, why, why. Like if the answer doesn’t solve the solution, I’ll just keep going for the solution.

Let’s start at the big one for me with testing and the way that development, solutions, and architecture, everything, has evolved, that testing isn’t what it used to be. It’s not the– developers will produce code and, in comparison to modern day standards now, it was awful. And you would need that that testing was absolutely required to make sure that it functionally worked, and that’s now not the thing. We’re now in the state of play where people are like unit tests, question mark, like do we even need unit tests? Because the software quality is so high that we’re now kind of moving out and up closer towards the kind of UI end of things. So it’s completely transformed the kind of like what testing actually means and the test engineer or just tester doesn’t really exist anymore.

It’s now more like quality assurance, and it’s very generalized. You can use quality assurance in any industry, any role, and you can have a decent conversation for 95% of it and not need to be in the same industry because you’re there to make sure that you’re being an ambassador for effectively doing the right thing. You’re like the guardian of the user experience. A lot of it will be baking in quality right at the start, so making sure that you have quality assurance, not only you have an actual person that is regarded as quality assurance, but have it there as an ethos with everybody.

So you have things like the quality narrative, what’s the quality narrative within your company. And it’s like, who owns quality? Everybody does, right? If you want it to be fast and cheap and really high standard, it is possible, just everybody needs to be thinking the right way. So you’re not only there to catch the human error right at the start, so static testing guess, but you’re there to kind of beat it into people really, so that when the documents then reproduced or the next program or project is started, the level of quality is just higher even in the documentation process. It’s like it’s the same sort of idea of like testing was a big thing and unit tests were huge. And now, it’s the opposite way around. So same sort of thing, we’re building quality that you can focus on improving the customer experience, rather than efficiency in lines of code.

CARTY: Right. And you touched on a few things there that I want to ask you about a little bit later on. But just because we should test software doesn’t mean that we should blindly test everything, the way that it was done before. So how can organizations make sense of what does or does not deliver value when it comes to testing?

MORTLOCK: See, I probably touched on this. Loads of times that I talk about this, but it’s all about the results and to kind of, again, talk about it again because it’s quite an interesting topic. It’s pulling people’s safety blankets away, in terms of getting rid of unit testing. How many times have you had a bug that would have gone into production that was caught by unit tests? It may take, what, 45 seconds to run and a couple of minutes to write, but extrapolate that across the lifetime of a project and whatnot, it’s pointless.

You should be making sure that you are– just the way some of the bigger tech companies have automatic deletion of dead bits of code, same happens with not only test cases and regression cycles and regression suites, but techniques as well. Sometimes techniques will die out and you need to repurpose them or remove them. So it’s about making sure that you’re not falling into the trap of just because it’s quick means it’s cheap and, therefore just do it. Squeeze every penny. Be really, really efficient with it.

And if it’s the pesticide paradox, right, if you keep doing the same thing over and over and over again, and the bugs are going to get resistant, if you’re talking about real life. But you’re not going to find anything. All the bugs will be where you’re not testing. So if you focus on– that’s where you’ve built the quality really, really high– if you focus on that all the time, the bugs will be outside of this area. So stop testing there, start testing somewhere else. And if you start to see a re-emergence of issues and bugs, then that’s where you have to be flexible and like re-implement new strategies and increase, kind of quality assurance in that area. So you don’t need to test.

The testing, in theory, should just be– testers in theory, shouldn’t exist. You should be redundant because the quality of the software is so high that you don’t need to test anymore, right. That’s the idea. We’re there as kind of like a backstop. It’s not the silver bullet that everybody thinks it is. The silver bullet is better quality, it’s not more testing.

CARTY: Absolutely. And that’s an interesting idea, talking about unit tests being redundant. And you hit on that a little bit already. Are there other QA techniques or approaches that organizations should take a little bit of a deeper look at in terms of the value that they deliver, and whether or not those are redundant as well?

MORTLOCK: Yes. For every company. Even the ones that are like best in class, and even the ones that are just starting up and don’t really have any quality assurance, it’s just developer production. But it’s like– it’s going to be one of those cliches, but it’s personal to the company in terms of where the quality has been built. And it’s where you’re not seeing issues. This reminds me very much of the– I’m not sure if you’ve seen the diagram of the bombers coming back from the war and it’s showing you where they’ve been hit. And they’ve said, right, we need to put armor there. It’s like, no, those ones are all the ones that are actually coming back and surviving. Maybe we should put the armor where they’re not being hit because they’re the ones that aren’t coming back. That’s obviously the weak point. We’re not seeing the ones that crash because they don’t come back. So that must be where the issues are.

So that’s what we should be doing. Using data that’s coming back from either testing or production to then be specific in the way that we’re armoring our bombers as it were. Because otherwise, you just get too heavy. You can’t fly, you can’t fly as fast, and you’re effectively hamstringing yourself by kind of over protecting or over testing as it were.

CARTY: This idea of adhering to your software quality narrative ultimately has a positive impact on everybody, right? I mean, you’re all about efficient processes and a thoughtful approach to how you spend your budget, things like this. It’s nice to save the business some money, but an efficient approach benefits everybody, right?

MORTLOCK: Definitely. So the big one is– or the easiest one to get past the shareholders and seniors is money. It’s cash at the end of the day, isn’t it? If you can make efficiencies in costs, you’re going to probably get stuff signed off. But with that comes– like say you’ve got a test environment that’s just– like we’ve got a place called Spaghetti Junction, and it’s just a overlap and cross of roads and bridges and whatnot. If you’ve got an environment that’s like that, it’s going to be a nightmare being able to keep it in sync and your cycle time instead of being like a day will be like 10 days because the intricacies are so huge. You go through and rip that out and make it streamlined and efficient, you’re going to save the business days and days and days. But also, your developers and your QAs are going to be so much happier. Because being able to just test something or just deploy something is going to be so much easier. So you save the business loads of money, then all of your employees within development are way happier, therefore they’re more productive, therefore you make more money. So it’s like the infinite reward.

CARTY: And this idea, this outcome, hinges on that shared responsibility approach to digital quality, right. So understanding the why of software testing probably goes a long way toward getting the rest of the organization to buy in, which is what you started talking about earlier a little bit. Beyond that, what else can organizations do to foster this idea of a shared approach to digital quality, and why is that so important?

MORTLOCK: So long story short is it’s cheaper, for one. So if you get whoever writes your initial documentation, whether it’s a business requirement document or a user story or a Cucumber scenario, whatever it is, if that quality is higher, then therefore, it will extrapolate out to less bugs. And you catch all of the bugs– you go as far left as possible and catch as many bugs as you can because there’s QA engineers and developers with the right mindset. And software developers in test actually in the kind of sizing and shaping, you’re going to add more quality there. It’s almost free there because those people are being paid for. You remove that. And if everybody’s invested in quality, they’ll want to remove those. Whereas back in the day, you would just do your normal dev test, it would go into prod, you’d find all the bugs, you’d put them back in. That’s a mega, mega expensive way of doing it. That’s probably going to get everybody kind of behind it.

But again, it’s the it’s the happiness thing as well. You’re going to get happier people. You get less of the, oh, what is this coming through from the developer. Being like, oh, this doesn’t make any sense. This is impossible. It’s not going to be three story points, this is like 21 story points. It’s got to go back through. And then you get a developer that’s interpreted a piece of requirement that they haven’t seen yet, and/or that’s been written poorly. And it goes over to test and they’re like, I was in that refinement. This is not what they were talking about. We have to redo this. If you breed this quality mindset rather than just testing and unit test coverage and all that sort of stuff, you get it by being lean, right. Build quality in rather than kind of shoehorning it in the end and trying to manufacture quality.

CARTY: And this takes time, right? I mean, this takes a lot of time, a lot of effort. You have to win over a lot of hearts and minds. I’m sure that’s not always an easy battle, right?

MORTLOCK: Yeah, definitely. So the strategy– there’s lots of strategies to go about this. You can be like a belligerent warlord and just battle people until they just surrender, I guess, or you can–

CARTY: I’m guessing that’s not the way to do it, though. I’m guessing there’s a better way.

MORTLOCK: Well, it depends. It depends how great you want to be, I guess, and how long you’ve got. It’s one way. It’s like hammering a screw into a wall. You can do it, it just won’t be pretty. It will probably be able to hold a picture up but I wouldn’t want to put my TV on it, let’s put it that way. And you’ll make an awful mess.

So I took– so I went through a transformation process with my company, and a secondary one as well, in terms of environments. And taking a sales strategy or a pervasive marketing strategy on this is very advantageous. Where you go for the early adopters, that’s the best way to do it. You want to incite pervasive excitement, you go to the people that want to do it. You go to them first and you rally them up and you pep them up, and then you watch the seniors who weren’t involved say, about all this excitement and I wasn’t involved and this thing sounds like something that I should be behind and I should be the sponsor of this. And it’s pervasive that way. If you try to battle through the people who don’t want to do it, you’re going to have a really tough time. And then the propaganda that comes out of that is going to be a bit lackluster anyway. Whereas if you go for the people that are there for quality, they’re there because they want to change because make money or make happy or whatever, that pervasive happiness will be, yeah, exponential. That’s the way I’d go about it. Not just because it’s easier. It’s just more productive in the long run, for sure.

CARTY: And at the end of the day too, it’s really all about delivering a product that the customer not only wants but will want to use and there’s a differentiation there. How can organizations better prioritize the customer’s needs as part of their digital quality strategy?

MORTLOCK: Yeah, I guess this is an interesting one. I’ve spoken about this, I guess, a couple of times, where you’ve got to remember you’re not delivering code. Code is the enabler to customer experience. So you’re delivering– like people say, oh, yeah, it’s just a user story. You’ve got to remember it’s like concentrate. It is a user story. It’s a story, right. And it’s got to make sense. It’s not– as a product owner, I want this field to equal that so that this will then have that output. That’s not a story, that’s just a requirement. That’s kind of– you’re kind of missing the point of writing things that way. If you’re going to do that, go back to just writing requirements. You want a story of a user, and you want to be inclusive of journeys and see how people experience the code as it were. You need to be kind of thinking like that. You’re not a work item machine, or a code delivery machine, or a user story machine. You are delivering user stories or user journeys that create customer benefit and business benefit. You need to stop thinking like an engineer and start thinking like a customer. That’s the big one. We’ve spent all this time trying to make sure that the code is as great as possible and super efficient. And really, really jazzy with all the new JavaScript and Node functions and whatnot. That’s great. We’ve got there. We can quit that now. Let’s have a look at the user experience shall we?

CARTY: James, in one sentence, what does digital quality mean to you?

MORTLOCK: I would describe it like– I remember the first way someone described punk rock to me. It’s not just a genre, it’s a way of life. And it’s very similar. It should be. It’s not just a title or it’s not just a thing that you say, it’s an ethos. You know, like people talk about DevOps or Lean or Kanban or whatever. You can’t just do it, you need to be it. And the company needs to be there. So yeah, it needs to be– almost, it needs to be like a feeling and a movement rather than like a process, because people tend to switch off from stuff like that. And people are going to need to want to do it. So digital quality is everybody wanting that customer experience at the end. And the employee experience, in terms of deployment and delivery, being really high quality and very low effort. That’s the gold isn’t it? We want to get to that. And digital quality helps enable that, in my opinion.

CARTY: I’ll have to follow up with some band recommendations or something like that, James. So stay tuned on that. But let’s go to the next question. What is one software testing trend that you find promising?

MORTLOCK: So we’re having this– I was having this discussion with our principal engineers. And one thing that I want– I’m desperate to talk to them about, and we’re going to dedicate an entire working group session to, is mutation testing.

So it’s kind of like– it’s when you can crawl and you figured that out, and then you can walk, yeah, you figured that out, you can run, yeah, and then there’s the sprinting. So don’t sprint before you can crawl. So we’re not quite there yet so we need to figure out the running bit. But mutation testing is, in my opinion, is the next best thing. And it’s kind of silly because it’s like testing whether your testing is testing properly. It shows the quality of your testing, so you’re purposefully mutating things and checking to see that your quality processes and your testing processes are robust. So it’s like, who will guard the guards? Like you’re making sure that the guards are guarding properly, and your tests are testing correctly. A mutation testing is kind of like an offshoot of chaos testing. Will check like the robustness of your quality systems. So I’m quite interested in that.

CARTY: And as we’ve talked about this, clearly a need a need for that introspection, right, and to evaluate your own processes. So that makes sense. James, what is your favorite app to use in your downtime?

MORTLOCK: I spend a lot of time in YouTube. Really, really– yeah, I’m on the beta for YouTube as well, and have been for a while. I’m testing even when I’m not testing. That’s quite sad, just thought of that.

That’s just someone that– or if I refer to them, if I’m to personify a company, that’s someone who knows what they’re doing in terms of, we want to deliver a customer experience, so what do we want them to experience? And it can have bugs, potentially. But it’s the fact that the thought that has gone in to enrich a small area of an application, that’s very important for me.

One of the earliest moments of that– that is just one of those– that is just excellent kind of moments is, in Slack, on the mobile app, if you used to be able to grab the top corner of a picture and then swipe, instead of the picture just falling down, it would spin based on the physics of where you’ve put your finger and where you’ve thrown it. It’s absolutely unnecessary, I don’t need that, but very much enjoy it. And it’s that level of detail that, I think, makes a good app. People can have P1s and P2s and you just say, yeah, it’s broken, or the back end is down. But it’s the P3s and the P4s that really make an app outstanding. And that sort of stuff that was obviously like a hackathon, like a Hack Day thing, that’s gone into prod. That’s what makes it great app, in my opinion.

CARTY: It’s the little touches sometimes. Just those little details go a long way.

MORTLOCK: Exactly.

CARTY: And James, finally, what is something you are hopeful for?

MORTLOCK: Oh, wow. Something that I’m hopeful for that– I’m hopeful for, I guess, two of the points that we’ve touched on today is that quality, like ethos that mindset, is like baked into everybody rather than it being trying to deliver things and deliver code. It’s about delivering customer experience and delivering it with quality. I’m hopeful for that. The momentum of that seems to be picking up because we’re getting into that area where the old techniques, software quality techniques, are becoming redundant because that area has just been. You’ve rounded that circle so much, it’s absolutely perfectly round, so you can move on to the next shape now. I think that’s a big one for me. Almost everything that we’ve spoken about, I think I’m quite hopeful that the tables are kind of turned from how much can we kind of plumb into development. It’s now, how much can we kind of invest in quality because it’s going to be cheaper in the long run?

CARTY: Well, James, this has been fun. Thank you so much for joining us.

MORTLOCK: Thank you very much. Thanks for having me. And hopefully everybody’s learned something or enjoyed it.