Don’t Be One of These Testers

Transcript

(This transcript has been edited for brevity.)

CARTY: Want to stump Kristin Jackvony? Good luck. Kristin loves puzzles and escape games. Really, any situation where she can use her wit and the tools available to her to solve a problem, she thrives.

JACKVONY: Somehow, I came across, I think it was called Doors, where basically, it’s all these little mini puzzles where you have to solve the puzzle to be able to open the door and go into the next room. And I was just hooked. I have probably played hundreds of these games now. So yeah, they’re a lot of fun.

And I think one of the things I like about them is that you know you have all of the tools that you need to solve the puzzle, as opposed to some of the puzzles that people like to solve that are sort of like lateral thinking, where you have to think outside the box and imagine things. You know all the tools are there. You just have to figure out how to put it together.

CARTY: Kristin has been interested in puzzles from a young age. But her journey into puzzles and escape games as an adult has led her to some interesting intellectual places, including an interest in a new language.

JACKVONY: It seems like there are a lot of escape games that are Japanese. And because of that, I actually started to get interested in the Japanese language. So I’ve been learning Japanese on Duolingo. And it’s fun now when I’m playing a Japanese escape game, usually, they will give you all of the clues in English. But sometimes, there will be like a sign that’s in Japanese. And it was really fun for me the first time that I could read that what the sign said was ramen. I’m like, I know what that says. So that was really exciting.

CARTY: It’s not puzzling to see why Kristin became a QA manager and tester. Just like an escape game, testers are always on the hunt for clues in an application, even if it’s not as simple as finding them behind a hidden bookcase door.

JACKVONY: The reason why I am a software tester and not a software developer is because I really enjoy being presented with sort of, not a finished product, because, of course, it’s not finished yet, but a product that all of the clues are there, I guess you would say, kind of like with the escape game. So somebody says here, we’ve coded this new feature. We need you to test it.

And a lot of times in my career, I’ve discovered that even though we’re supposed to have documentation about how features work, a lot of times, there’s no documentation. So it takes a lot of experimenting, a lot of playing around. And I really enjoy that. I really enjoy the exploration part of trying to figure out how a feature works and then thinking, what are some ways that I could test this? Or what are some edge cases that we might need to explore? And then thinking, are there any ways that I could break this? I think that’s really, really fun.

And I think that’s different from what developers do. And I appreciate developers so much, because they are the ones that are building the things. They are starting with nothing. They’re starting with a blank sheet, with no code on it, and trying to figure out, I need to build this feature. How am I going to do that? So that’s really an admirable skill. But my skill set is more exploratory, figuring out how something works, figuring out all the ways that it might not work, and then trying those out.

CARTY: This is the Ready, Test, Go. podcast, brought to you by Applause. I am David Carty. Today’s guest is master puzzle solver and test engineer, Kristin Jackvony. Kristin is an experienced QA manager and tester, currently serving as the principal engineer in software testing at Paylocity. She organizes systems and processes to support all areas of testing, whether that’s for improving legacy software or supporting new software from the early stages of development. Kristin also speaks at conferences and is the author of the book The Complete Software Tester, Concepts Skills and Strategies for High-Quality Testing. So, what does it take to be a complete software tester? Kristin would know.

Kristin, first of all, congratulations on your book, The Complete Software Tester. So from a high-level perspective, what should a complete software tester look like today, particularly as it applies to their skills and their competencies?

JACKVONY: Well, the most important thing — and this is often sadly overlooked — is they need to be able to find bugs. That’s why we are software testers. We are trying to find the bugs before the software is released and the customers find the bugs. So that’s absolutely the number one thing.

Another thing that is very important, especially these days, is to understand APIs and how APIs work and how to test them. So that’s very important.

And then I would say the next level is test automation. You need to be able to automate. We cannot, at the rate that software is being developed today, we cannot get by on just manual testing. We need test automation.

And then finally, it’s very important for software testers to understand some of the adjacent software testing areas, such as security testing, accessibility testing, performance testing. These are all very important. And they are becoming more and more important each year.

CARTY: Right. Now, you’ve discussed and you’ve written about six testing personas to avoid, right? So today, we’re going to talk about two in particular, Job Security Jim, and Conference Connie. I keep wanting to say Jim and Pam, but it’s Jim and Connie. So can you walk us through what those two personas look like?

JACKVONY: Sure. So Job Security Jim is a guy who’s been at his company for a really long time. And because he’s been there so long, he understands the application backwards and forwards. He knows exactly how to configure things and how things are supposed to work, which is great. But if the company starts to evolve, as are all companies that need to survive do, and they might have a rewrite of the application and in a new coding language, or they might be making some changes, Job Security Jim sometimes feels like he doesn’t have to really learn that. He doesn’t have to learn automated testing, or he doesn’t have to learn automated testing with a new tool. So what winds up happening is he’s stuck sort of in one place with his one skill set. And that’s very dangerous, because as the company evolves, he’s going to wind up getting left behind.

And then Conference Connie, she’s the one who’s very, very excited about new things. She loves to go to conferences. She loves to find new ideas, learn about new tools. And all of that is really wonderful. But Conference Connie will often come back to her workplace and not do anything about the new tools that she learned about or new techniques. Or sometimes — this is sort of a variation of Conference Connie — she might go back to her workplace and say, we need to scrap everything that we’ve done so far with our automation and start all over again with this new tool. So both of those things are very disruptive. And both of those things are — Connie is learning, but she’s not learning deeply. Connie is not learning in such a way that she’s actually going to be able to benefit her team.

So the thing that Jim and Connie have in common is both of them are not learning and growing.

CARTY: Right. It’s interesting, because they are diametrically opposed in some ways. But they actually have some similarities here. So to go back to Job Security Jim, this person brings a lot to the team, right? I mean, they have some deep domain knowledge that can be really useful in that respect. But there’s also some negatives. So can we dig into a little bit more of what they bring to the team with respect to what they offer their fellow team members, and what some of their weaknesses are there?

JACKVONY: Yeah. So the positives of Job Security Jim are that he knows where all the bodies are buried, as it were. So he understands all the little tips and tricks for if something’s not working correctly, or what those bugs that have been sitting around forever and nobody has bothered to fix, and how to work around them. He might also have knowledge about how to configure complicated testing situations. So those are all real positives.

But the negatives are, as he falls behind, as his skill set doesn’t improve, while what’s needed for the job becomes more complex or more advanced, he’s going to start to become a bottleneck. He’s going to be the guy that is continuing to do the old manual testing or using the old slow tool, while everybody else is waiting for him to get his work done so that they can continue to move forward.

CARTY: Right. And you said test automation is one of those skills that a complete software tester should have. It’s one of the things we talked about right off the bat. So as time goes on, more testers will be expected to embrace automated testing. So this is an area where Job Security Jim in particular, is likely to get left behind, right?

JACKVONY: Oh, absolutely. Yeah. Without question. I mean, there are few jobs left today that require manual testers. But there are very few. They don’t pay well. They don’t allow for good career advancement. So you’ve got to learn how to automate. I mean, you’re just as smart as everybody else is, speaking to the audience here. And if other people can learn how to do it, so can you. So you got to start.

CARTY: Right. Now, I’ve done the conference circuit before, as have you. It can be a great source of inspiration and ideas, that kind of thing. But that’s some of the challenge. How can the Conference Connie persona get a little bit more focused or a little bit more grounded to come back to their team or their organization with some real concrete, actionable ideas?

JACKVONY: Well, I’ve got a couple of ideas about that. So one idea would be that Conference Connie should think about what problems she’s trying to solve before she goes to the conference. So let’s say, for example, on her team, they realize, oh, we’re struggling with our mobile automation. We want to get better at mobile automation. So she could go to that conference and sign up for some of the presentations that are going to help her answer that question. So then she can focus, and she can start making a list of actionable information that she can take back to the team.

Another thing that she could do, if there’s no particular problem that the team needs to solve, she can go and while she’s attending the sessions and taking notes, she can be thinking about, what is an actionable step I could do when I get back to the team about this one thing that I’m learning? And because she’s so excited, she’s probably going to come up with 20 different actionable steps. But what she needs to do is maybe on the plane ride home, she needs to be looking at, what did she have on her list, and picking maybe just one or two things to take back to the team.

And definitely, it’s not a great idea to go back and say, we need to completely rewrite our automation, because that’s just going to stress everyone out. So don’t do that. If there’s a real problem with the automation, then maybe you might want to say, I’m going to do a proof of concept on this one new tool. And then we’ll discuss it afterwards. Something like that’s okay.

CARTY: Right. Right. And to that point about stressing out your fellow team members — Conference Connie can be perceived as a dog chasing cars, right? There’s always another bright, shiny one coming around the corner. So if you are already Conference Connie with all of these grand ideas you come back with, but you’re trying to be more grounded, you’re really trying to actually make these things actionable, what can she do to sort of fight that negative stigma that might be attached to him or her?

JACKVONY: I think she needs to start building some trust. So when she comes back — you know, she’s going to come back. She’s going to be excited. She’s going to be filled with ideas. I would encourage her to maybe write a blog post about it, like if the company has maybe a place where they can share those things, or just write a document. But then when she meets with the team, just have one idea. Like, ‘Hey, everybody, do you remember that problem we were having where we were having trouble figuring out how to parallelize our testing? Well, I learned this one tip that I think will help us.’ And have it be something that the team can adopt without a great deal of pain and stress.

And so when the team adopts that, then they’re going to say, ‘Hey, you know, Conference Connie’s got some good ideas. So maybe we’ll listen to her a little more next time,’ instead of ‘Oh, no. Here we go again. She’s going to come back with 10 things that we can’t use.’ So I think it’s just slowly building up trust that way.

CARTY: Right. That makes sense. Now the stark difference between these two personas, it speaks to that DevOps-y kind of idea of evangelizing knowledge. And that all sounds good until you’ve got these two very, very different types of people. One is resistant to change, but really digs in deep and absorbs information on a detailed level. The other loves the idea of change, but might struggle to implement it or kind of get to that next level or even understand it in a concrete way sometimes. Is that how you see it? And how can managers kind of affect change with these two personas and evangelize knowledge in a way that will really stick?

JACKVONY: Oh, that’s a great question. And I think that that’s something that a lot of companies struggle with, because there’s always going to be a tension between, we need to grow and evolve and change, and we need to keep things the way they are. And there’s cases to be made for both sides of those things.

I mean, leaving things the way they are ensures that your software is going to continue to chug along nicely the way everybody’s expecting. But making sure that you’re adapting and growing means that your software is going to adapt and grow and provide more value, which is what your customers are going to expect.

So I think the most important thing to do is to strike a balance between the two and make small, incremental changes, evaluate those changes, and then come up with what the next step is and continue to move forward.

And I think it’s also very important, and there are a lot of teams, especially managers like to do this, they like to come up with the big idea and expect that everybody’s going to pivot to the big idea. But then, they get distracted. They think about something else. And then the big idea is lost. And those who are trying to implement it are stuck with, well, ‘What do we do now? Do we keep trying to implement the big idea? Or do we go back to the way things were?’ So I think slow and consistent progress is the best strategy.

CARTY: Right. And probably a case to be made for keeping things the same and implementing change at the same time, depending on the project, or depending on the team even. So it definitely speaks to that what you’re talking about striking that balance. So when these personas hit the job market, what are some immediate red flags that you can spot? Are there items on their resume, or things lacking from their resume that really stick out to you right away?

JACKVONY: Yeah, I think there’s two different things that I look for that are potential red flags. And one is the job history.

So, Job Security Jim might have been at his company for 20 years. And sometimes, that’s really wonderful because that means that he found a good place, that he was making a difference, that people were valuing him. But it can also mean that he got into a job that he thought was cushy, and he just dug in his heels and he stayed there. So think about that. If you’re a person who’s been with your company for a really, really long time, you’re going to want to make sure that you’re showing some movement. What kinds of projects did you work on? What kinds of new things did you learn?

So, then, at the opposite end, Conference Connie can sometimes be a bit of a job hopper, because she gets really excited about whatever the new tool is. Oh, there’s this new startup, and they’re using the new testing platform. So I want to get on to that. So if I see somebody who’s hopping jobs every year for several years in a row, that’s a red flag to me because that says, if I hire this person, they’re going to get bored. And they’re going to want to move on very quickly. So to be able to show that projects were completed, to be able to speak to why it was that you left one job for another, that would be very helpful.

And then the other thing that I look for on resumes is also skills. What skills do these people have? And so if I see Job Security Jim is using one antiquated tool that no other companies are using, that’s a red flag for me because I think Job Security Jim hasn’t bothered to learn, let’s say, Node.js. And we use Node.js all the time. So how do we know the Job Security Jim’s going to be able to transfer his skills to our team?

But, then, Conference Connie can sometimes tend to put every single language or tool that she’s ever been exposed to, even a tiny bit, onto the resume. So then it looks like she is sort of jack of all trades, master of none. And that’s concerning too, because at my company, we’re going to have specific tools that we use. We’re going to want Connie to be able to really dive into those tools and use those. And if she only has a limited knowledge of everything, we’re not so sure how well she’s going to learn. So think about when you’re including tools on your resume, think about what projects you used those tools for. So make sure that you have that kind of experience on your resume that will help ease the hiring manager’s mind.

CARTY: Right. And the workforce, the workplace has changed so much in the last few years. I’m putting you on the spot a little bit with this question. But are there any emerging personas that you are researching or writing about in today’s landscape that reflect the changes that have happened in the world?

JACKVONY: Well, I think one of the personas that I list in my book is Automation Annie. And this isn’t really new. But I think the problem is continuing to get worse. We are continuing to see ‘software testers’ — and I’m using that in quotes — who really what they know how to do is write test automation. Test automation is great. But if you don’t understand your application, and you don’t understand what it is that you should be testing for, really, all you’re doing is busywork. You’re writing code that might not provide any value at all.

And one of the things we do at my company when we are hiring software testers is, we give them a challenge where we give them a buggy application. And we say, please find the bugs and give us a bug report of all the bugs that you found. And it is astounding the number of applicants I have seen who have as much as 15 years of software testing experience who can’t find the bugs. But on paper, their resume looks wonderful, because look at all of the wonderful things they’ve automated. Look at the code they know, the tools they use. But if they can’t find the bugs, they’re useless to me. So I think that’s a disturbing trend. And it has continued to get worse.

CARTY: That’s interesting. In some cases, is it somebody that is struggling with the pressure of finding the bug in that particular situation? Or is it just a skills issue there?

JACKVONY: I think it could be a skills issue. I don’t think it’s a pressure issue because it’s not timed. The challenge that we give them is, take a few days. See what you can find. So I don’t think it’s that. I think for a lot of Automation Annies, I think they actually aren’t interested in finding the bugs. What they like is, they like to write the code to test the software. They find that fun.

And I think maybe, it’s sad to say, but I think there are some people who enjoy writing code, but they don’t want to be software developers because they don’t want to be held accountable for making a feature that then people can’t use. But if they write test automation that is flaky and has lots of false failures, they might not be held accountable, especially if they’re job hopping from place to place. So I think that’s what it is. I think it is a lack of skill set and in some, a lack of caring.

CARTY: Wow. That’s an interesting trend to keep an eye on. And it really speaks to the value of effective, exploratory testing, as much as we want to focus on automation. It really gets back to that point. So, Kristin, if there’s one blanket piece of advice that you could give to a tester who is really struggling to figure out the next step in their career, what would that advice be?

JACKVONY: Well, one piece of advice would be, read my book.

But, seriously, I think, don’t feel like you have to boil the ocean. You know, that’s the expression where you just take on too much all at once. I think sometimes testers can be hard on themselves, and they can feel like there’s so much that I don’t know. But all you need to do is find one area to work on, say to yourself, what is something that is a weakness that I have right now that I could learn, a skill set that I could improve on, that would have a big impact on my career?

And, so, for some testers, it may be automation. And, so, what they can do is, there are lots of great tools and programs that they can use to learn how to automate testing, including, I’ve got a whole section in my book about test automation and how to get started with test automation. So set a small goal for yourself. Maybe this month, I’m going to look at what my team has in test automation, and I’m going to try and make a change to one test to improve it, or something like that. Or if your team doesn’t have any test automation at all, you could say, I’m going to pick a test automation tool, and I’m going to do a little proof of concept and run one test, just small things like that.

Another thing that’s very helpful — and I got this tip from a coworker — is to actually put time on your schedule for professional development. So if you say to yourself, every Thursday at 3 p.m., I’m going to take an hour, and I’m going to work on improving my skills. And mark it off as a meeting for yourself. Nobody has to know that it’s just a meeting with yourself, so they won’t invite you to whatever other meeting is happening at that time. And then you can say, I’m busy. I’m in a meeting. And you can use that time to improve. So start small and be consistent.

CARTY: Right. And I know in your book, you were really purposeful about including some different things that the reader can work through, some different examples, right?

JACKVONY: Yes. In the automation section, I actually have a GitHub repos that you can pull down, and you can actually run those things on your own computer at home to see how they work. One of the ways that I really like to learn things, especially around coding is, I like to see how somebody else did it. And then I can transfer — I can say, ‘Oh, well, that’s the syntax to do such and such. So now, I can just copy that and put it in my own project.’

I’ve also created as part of this book, to go along with the book — although you don’t even need to buy the book to use it — there’s an application, a web app, with an underlying API that anybody can use to test. And they can test making API calls and then seeing the result of those calls if they go to the web app. And they can see how those two things work together. They can actually use it to practice writing test automation. So that’s a useful tool. And with the book, you can really kind of step through how to start to acquire those skills.

CARTY: Okay, Kristin, final sprint questions here. In one sentence, what does digital quality mean to you?

JACKVONY: To me, digital quality means applications that are providing value to their users and are doing so reliably.

CARTY: Perfect. What will digital experiences look like five years from now?

JACKVONY: I’m expecting that five years from now that people will have a more personalized experience with all of their applications.

One of the things that I’ve noticed over the last few months is that every time I get a new computer or get a new app on my phone, I’m being given the opportunity to choose light mode or dark mode. And I think that that’s only going to expand, that people are going to say, what color would you like the app to look like? Or which things would you like to see on the home screen? So I have a feeling that as we move forward, we’re going to see more and more of that personalization.

CARTY: I’m [on] team dark mode. So I hear you on that.

What is your favorite app to use in your downtime?

JACKVONY: So it was hard to choose. I have three, because I like to always have my brain going no matter what I’m doing.

So, if I’m running on the treadmill, or if I’m doing the dishes or folding the laundry, I like to have something going on in my brain. So I really, really enjoy Audible. I enjoy listening to books while I am walking or running. That just keeps me engaged and keeps me thinking about how much I’m suffering on the treadmill.

And, then, I also really like podcasts. I have an app called Podcast Republic. So I’ll listen to podcasts while I’m doing things like folding the laundry.

And, then, for while I’m working, I love Spotify. I like to listen to music while I’m working, especially if I’m doing like deep testing, I like to have that music kind of going on in the background. So all three of those. I’m not sure I could live without any of those.

CARTY: Sure. Totally understandable, and Ready, Test, Go. is available on a podcast player near you.

And final question for you here. What’s something that you are hopeful for?

JACKVONY: I am hopeful for more software quality. So I have noticed over the last several years that web applications have become more consistent, more consistently reliable. And that is really wonderful. I’m really happy about that. But every now and then, something will still surprise me, like I will be signing up for a new financial service, and I’ll discover that the form won’t take zip codes that have a leading zero, which basically leaves out all of the Northeast. So every now and then, I’ll find something like that.

And I feel like mobile applications are still a little bit behind. I think we’ve gotten used to the fact that, ‘Oh, sometimes the app will crash,’ or, ‘Oh, sometimes it won’t be responding, so I’ll have to kill it and open it up again.’ I would love that in the future we get to the point where that’s a real rarity. So I’m keeping my fingers crossed for that.