Finding Hidden Defects Using Investigative Testing

Do you find yourself executing test scripts repeatedly without finding previously unreported defects? This begs the question: If test execution doesn’t turn up defects, how do customers still find them?

Defect fatigue happens when testers execute the same tests repeatedly, whether automated or manual, without finding issues until customers report them post-release. Software testers must get creative and investigative to find defects hidden below the surface and between the integration cracks. They must learn to test beyond the happy path or primary application layers, and execute deeper tests that find hidden, often high-priority or critical defects.

This guide offers tips for finding well-hidden defects by mixing testing techniques with creativity. Make testing investigative, rather than repetitive, to stop defects from getting past QA and into the hands of customers.

What is defect fatigue?

Defect fatigue is similar to defect normalization, in which the software development organization directly or indirectly accepts the quality blind spots that exist within a product. Defect fatigue is a problem that software development organizations often completely ignore — or they might address it by assigning blame to development and test personnel.

For software testers, defect fatigue results from:

• repetitive testing with the same test scripts

• large numbers of existing, persistent defects, or technical debt

• loss of engagement or interest

• defect normalization within the development team

• test automation script errors or maintenance issues.

The majority of QA or software testing involves executing a test suite against new code builds to ensure existing functionality still works as expected. It’s a good thing. But, when all testing efforts revolve around repetitive tests, defect fatigue sets in over time. Combined with a persistent load of unfixed defects or high unresolved defect counts, defect fatigue begins to affect testing value. Manual testers become robotic and often disengage. When testers have years of experience testing the same application, they tend to skip over defects, especially when they know it’s a defect type that won’t ever get fixed. Skipping over visible defects out of habit or a need for speed results in defects escaping to customers.

Poorly maintained automated tests consistently fail over time. Automated test script failures require a great deal of time to evaluate, whether the issue is in the script or an actual defect in the application. It’s easy to miss some defects when they’re incorrectly labeled as script defects rather than application defects.

To mitigate defect fatigue and defect normalization, organizations can add testing that uses creativity and investigative techniques.

What is investigative testing?

Software testers use investigative tests with the intention of finding defects by breaking functionality. Investigative testing uses various existing manual and automated testing techniques without repetition. Investigative testing involves either manual or automated scripts that don’t persist or aren’t saved for repetitive use, unless they find a valid, high-priority defect. Tests are created on the fly and fresh for each cycle, similar to exploratory testing but with greater depth than standard regression or smoke testing suites.

Freshly created tests uncover defects by incorporating known code changes from defect fixes and new features based on testers’ expertise. QA teams can choose to document the tests or simply run them. If the team prefers automation, it should create investigative automated tests the same way: create the script, run it and, if no defects are discovered, delete it.

The goal is to intentionally locate defects by finding actions that break the application functionality. Make waves, and rock the proverbial boat! Here’s how testers can accomplish this goal.

Exploratory testing. Poke and prod the application. What unintentional functions exist?

Test connectivity. Validate APIs, database connections and third-party integrated software. Discover how connections can be interrupted or broken.

Negative testing. This includes inputting unique character types or special characters, as well as assessing boundary values.

Configuration settings. Customizable user settings tend to yield a wealth of defects, as they introduce uncommon user paths and experiences.

Security. See if you can circumvent security features, which often result in very high-value defects.

User navigation. Use the infamous browser Back button among other options, especially if the product instructs you not to do so.

Tips for investigative testing

Investigative testing derives its power from software tester creativity and a willingness to simply destroy applications. Test to destroy, rather than to confirm.

QA testing teams employ investigative testing using familiar techniques for experienced testers, including:

• creating mind maps to find testing areas to investigate, gaps in integrated connections or requirements;

• forcing the application to function outside the happy path;

• triggering unexpected errors to discover missing error messaging paths;

• exercising back-end processing and third-party software integrations to see what can be interrupted or failed by unexpected user actions.

Mind mapping creates an outline or checklist of features and functions to test. A mind map can be as simple as a drawing on paper that defines every known path in the application. Include configuration settings, customizable options and different data types in the mind map. Entering invalid date formats, for example, reliably yields a defect or two — map that. Now, once the known paths are mapped, move on to all the possible paths that are not known. Discover options hidden in the code or buried within the application, even if they aren’t frequently used.

Next, plan to test outside the expected results. For example:

• find non-logical user paths through the application

• click buttons more than once

• use all the buttons on a browser in sequence

• click repeatedly.

Users often click buttons in quick succession multiple times. This sounds silly, but clicking buttons in a variety of patterns often results in an application crash.

Use your personal experiences with applications to inform where defects might occur. How and where did you encounter a defect? Try it on the application under test, and see if it’s a common issue. Rely on existing experience and application knowledge to break application functionality.

Consider the application and all its integrated connections — APIs, databases, messaging systems and third-party software. Test and discover if API connections deliver invalid data, saving to the database hangs or failures in integrated software cause the application to crash. Look for steps that cause user documents, forms and other input data to be removed or disappear unexpectedly. Third-party software often executes functions such as sending email, SMS or other communications to users. See what user actions create bad messages or message system failures.

If an application supports different user roles, explore the application from these different perspectives and their respective settings. Can users bypass access settings or alter another user’s record? Take time to test these gaps as much as possible, as integration gaps are typically laden with defects.

Don’t forget installation defects or bugs generated after software upgrades. Many times customers create customized files for reports or forms. When an upgrade occurs, those files are overwritten. Application code must understand when configuration files have been altered and not overwrite customer changes. What software install or upgrade issues have you experienced? User experiences are key to finding similar bugs in multiple applications.

Browser developer tools

When running browser-based application testing, keep the developer tool logs open where they are visible. Developer tools exist for every browser, and you can typically navigate to them from the menu

Developer tools for Chrome consist of logs on the following tabs:

• Elements

• Console

• Sources

• Network

• Performance

• Memory

• Application

• Security

Console, Network and Elements provide logging functions for errors not always visible in the application UI. Within the Elements tab, users can drill into the code and view the properties of various elements. QA testers might try editing elements to invalid states and then checking the application response. The Console tab shows a variety of errors the user would otherwise not see in the application; the same applies to the Network tab. Developer tools provide easy access to view various log files during testing.

Behavior analysis to find defects

Finally, understand the behaviors of each member of the software development team. Developers, product managers and even documentation personnel all develop tendencies as they gain expertise in an application. Analyze the behavioral tendencies of team members and understand how each person operates to discover additional avenues for defects.

For example, many developers test their code on a local machine instead of a development or test server environment. The purpose is to isolate code to ensure it works as expected in optimal conditions and deliver it to QA for testing more rapidly. Often, once merged into a test environment, defects appear based on variances in the environment.

Many developers pre-test code on the test environment or create unit tests, while others might only do code reviews and hope the team catches any bugs. When developers don’t ensure code functions at a basic level, defects are typically easy to find. More developer testing makes it more difficult to find defects, but not impossible. People tend to make similar mistakes over time based on work habits. Consider this as you look for unexpected defects.

In a similar manner, product owners or managers tend to create user stories and requirements in the same way. Many tend to leave out the relevant workflow steps, roles, configuration settings and security access options. Think about it, how many product managers understand how the program functions from end to end? Should they? Yes. Do they? Maybe not. Track how each product manager creates requirements or acceptance criteria; you’ll likely find a repetitive type of defect created unintentionally.

Finding hidden defects requires testing against the grain rather than verifying a function performs as expected. Move beyond the happy path using familiar testing techniques to keep engaged in improving software quality and find hidden defects. The same path customers use to find often critical defects post-release.

Where possible, break out of the groupthink with seasoned, vetted crowdtesters. The Applause community of digital experts is available within hours to help fill gaps for new product launches, updates or any other testing need. Define the scope, market, demographic and devices, and Applause will source testers to meet your requirements. Learn more about Applause manual functional and automated testing solutions.