Crowdsourced Software Testing FAQs

No matter the technical expertise and attention to detail you have in hand, defects will always escape through to production. Today’s digital products and systems are simply too nuanced, and used in too many different ways, for all users to have a perfect experience with the product — even though that should be the goal.

Various software testing methods help organizations catch as many defects as possible. Crowdsourced software testing, also called crowdtesting, is one such technique that enables organizations to tap into a world of resources and experts beyond its own walls. For this reason, crowdsourced software testing can be a real differentiator in a competitive business landscape.

How does crowdsourced software testing work, and how is it implemented in the real world? We answer several frequently asked questions around crowdsourced software testing to give you a look into this powerful yet nimble approach to digital quality.

What is crowdsourced software testing and how does it work?

Crowdsourced software testing is an approach to digital quality that involves scaling testing outside of a traditional internal team. With crowdsourced software testing, an organization solicits real-world customers to validate functionality and provide feedback on apps or digital products being developed, typically through a crowdsourced software testing provider like Applause.

Crowdsourced software testing can occur throughout the software development life cycle, as long as a product can function on a basic level for crowdtesters to interact with it. A crowdtesting provider typically solicits individuals for crowdsourced software testing that fits the organization’s criteria. The organization might require testers that fit specific criteria, such as:

• ages

• demographics

• levels of education

• devices or appliances

• account status

• network carriers or conditions

• locations

• technical ability

• individual abilities, competencies or sensibilities

• physical characteristics

• languages spoken

• payment instruments

Crowdtesters typically follow basic instruction from the organization and the provider to follow the necessary user flows, document defects and provide relevant feedback on the product under test. Crowdtesters document defects for the organization to remediate before the product is released to a broad user base.

What are the benefits of crowdsourced testing compared to traditional testing methods?

Both traditional software testing methods and crowdsourced software testing have individual strengths. It’s important to take advantage of both internal and crowdsourced testing resources in software launches — not to look at one as superior to another. The approaches are not mutually exclusive.

Traditional testing methods will catch many defects, including those caught before code is committed to the main branch (pre-staging defects). Internal testers will catch quite a few more defects through traditional functional testing methods, including exploratory testing.

Crowdsourced testing attempts to catch the defects that fall into an internal development organization’s blind spots. These defects might include those that are only present in a production environment, localization issues, unique to certain device/OS combinations or other functionality and usability issues. Thus, the primary benefit to crowdsourced testing is that it gives an organization greater testing coverage — beyond what it can possibly achieve in house.

Additional benefits of crowdsourced software testing include the ability to scale testers on demand, customization for different projects or needs and reduced overhead of internal testing efforts. For these reasons, many organizations are able to achieve significant ROI with a crowdsourced testing approach, not to mention improving their company reputation and customer satisfaction.

How can you design and manage a crowdsourced testing campaign?

Managing crowdsourced testers is very challenging in house. Some larger companies, such as Apple, have beta programs that enable customers to volunteer as beta testers for new products or services. However, beta testing lacks the nuance and customization of crowdsourced software testing, and many businesses will struggle to source and manage both the testers and the in-house support teams necessary to succeed with the approach.

For this reason, most organizations turn to a crowdsourced software testing provider, like Applause, that can source a deep base of well-credentialed individuals, available around the globe with a vast range of device/OS combinations, skills and characteristics. Once a plan is in place, and a crowdtesting provider is engaged, the organization must clearly define test cases, including edge cases, and how to test them. As testing progresses, the organization and the crowdtesting partner should maintain effective communication between each other and crowdtesters as needed, to ensure results match expectations.

Once the crowdsourced testing results are in, the organization must collect, analyze and organize the findings for quick triage. Artifacts from crowdtesters might provide evidence of defects, such as a video displaying the issue, and the organization should manage these as well. From there, the organization can iterate both on the product itself and on the crowdtesting approach, working to improve development and QA processes for the next sprint.

How can you recruit and work with a diverse group of testers from different backgrounds and skill levels?

Simply put, managing a diverse pool of crowdsourced testers is an impossible task for most organizations. Crowdtesters must be representative of all the business’ customers — that means all relevant devices, account statuses, demographics, markets, languages and more. Think about the magnitude of the recruiting task, resource cost to maintain software testing managers and infrastructure support to document it all. It’s too much for most businesses, even highly successful ones — let alone managing all of that as economic, cultural and technical landscapes evolve.

Crowdtesting providers like Applause are in the business of recruiting these crowdtesters with different backgrounds and skill levels for companies in a variety of different industries. Crowdtesting providers don’t just work as an intermediary; they work alongside the organization to ensure digital quality goals are met — sometimes even working on-site on a daily basis.

Recruiting crowdtesters with diverse backgrounds and skill levels requires an ongoing effort. Communication with the crowdtesting community must occur on a regular basis, even among testers who are not currently active, to source the kinds of digital experts that businesses require, now and in the future.

What are the key metrics and KPIs for measuring the success of crowdsourced testing?

When done correctly, crowdsourced testing can deliver or exceed the organization’s established goals. Key metrics and key performance indicators (KPIs) for crowdsourced testing will vary between iterations, projects, teams, businesses and industries.

So, how can you measure the impact of crowdsourced testing on product quality and the user experience? Some general success indicators can help you gauge the effective of your crowdsourced and internal testing efforts:

• Defect severity analysis. Categorizes how many bugs of each priority level are reported, such as critical, major and minor defects.

• Defect detection rate. Calculated as accepted bugs divided by total bugs reported, this metric shows how well crowdtesters stay within the scope of the testing.

• Qualitative/quantitative UX feedback. Real-world testers can provide useful usability insights, whether conducting a full-fledged study or simply augmenting internal UX efforts.

• Cost per defect. Calculates the cost-effectiveness of the crowdsourced testing engagement – the lower the number, the more efficient the testing.

• Crowdtesting vs. added internal budget. Measure the amount of budget spent on crowdtesters compared to what it would cost internally to test. This comparison can also extend to travel costs if those internal testers needed to validate functionality in a specific location. Likewise, compare the crowdtesting spend to the cost of devices or device labs required for testing.

Don’t forget that crowdtesting might also influence broader testing metrics and KPIs, but are only part of the overall digital quality picture. Some of these testing KPIs include:

• Test coverage. This percentage of total features tested is an indicator of how comprehensively the organization tests the application.

• Defect turnaround time. This measures the amount of time it takes to resolve a defect once it is submitted, and it can be measured relative to crowdtesting defects or overall defects.

• Retest rate. Calculated as defects requiring a rest divided by total defects, this testing metric explains how efficiently bugs are remediated.

How can you validate and incorporate feedback from crowdsourced testing into the development process?

Crowdsourced testing can occur throughout the software development life cycle to identify defects in digital products. Whether validating a prototype or a live, publicly available product, the goal is the same: to identify problems for developers to remediate.

To ensure valid, valuable feedback from crowdtesters, the organization must define the testing scope and parameters. A crowdtesting provider like Applause can communicate these criteria to testers. Crowdtesters should follow bug report best practices, including clear titles, thorough details, and supporting evidence, such as a screen recording showing the issue.

Try to find a crowdsourced software testing partner that integrates seamlessly with your development workflows to ensure feedback is tracked in a useful way, such as through Jira. Developers can attempt to recreate and validate these defects internally, but that might be difficult or impossible, as crowdtesters are often validating flows that cannot be tested internally.

How do we ensure the security and confidentiality of software and data during crowdsourced testing?

As with any vendor integration, organizations should establish firm policies that protect the business from any security risks associated with crowdsourced testing. Some ways organizations can protect themselves (and their customers’ data) is by using crowdsourced software testing vendors that adhere to standards like:

• SOC 2 Type 2

• GDPR

• Privacy Shield

• California Privacy Rights Act

Whether you recruit crowdtesters or select a third-party vendor to handle that task, both approaches require firm access control. Some ways to manage and streamline access as well as boost security include:

• user account administration

• access management functionality

• user roles with pre-defined permissions

• limiting access to production systems to operations personnel

• software infrastructure protection, such as threat detection and firewalls

• physical server security

• third-party integrations with only compliant organizations

Many crowdsourced software testing partners like Applause include a boilerplate code of conduct and terms and conditions that prohibit crowdtesters from discussing the nature or details of the product or their testing activities. Organizations can also opt to require an additional non-disclosure agreement (NDA) covering additional parameters.

The organization or the crowdtesting vendor typically limits access to the information crowdtesters receive. Project requirements and product capabilities help define the crowdtester limitations.

What are the best practices for crowdsourced testing?

The best practices for your organization in its crowdsourced software testing journey will likely depend on its maturity with the approach. For organizations that are just getting started with crowdtesting, the goals are often small and incremental. Bear in mind also that crowdsourced software testing best practices might also vary by industry, solution area and current — yet evolving – business goals.

In a webinar called Onboarding a Crowdtesting Partner, Applause CTO Rob Mason offered the following keys for success:

• Determine which problems to solve. Crowdtesting helps solve a variety of digital quality challenges, but only if you can grasp the problem areas in the first place. Whether the goal is to improve test coverage, accelerate your delivery speed, gather real-world insights on products or test against internal blind spots, it all starts with a clear understanding of what you hope to achieve and how the crowdtesting partner will enable those results.

• Choose a pilot project. Select a project with a narrow focus, clear scope and short timeline. These clear, concise criteria will enable the crowdtesting partner to work quickly and effectively to deliver results for the business to analyze. And a pilot allows the business to evaluate the effectiveness of the partner without a commitment for more work.

• Define success criteria. Establish what you hope to achieve with crowdtesting and the metrics that will measure success. Whether the goal is to reach a specific percentage of test coverage, see a certain level of ROI or establish a community of testers for long-term use, it’s important to establish those goals alongside the crowdtesting provider.

• Communicate often. Effective crowdtesting partners share in your successes and failures. Discuss your goals, concerns, needs and wishes with the crowdtesting provider so they can deliver effective results.

• Learn and grow. After a successful pilot, use those learnings to establish clearer goals. Each crowdtesting engagement offers the potential for improvement — that’s the beauty of crowdtesting; you can shape it to what you need. Think iteratively, and improve.

Organizations with mature, comprehensive, adaptive crowdtesting engagements will find that their practices and goals change over time as they drive operational efficiency and maximize ROI. Don’t forget that other team members or departments might be unaware of crowdtesting’s potential to help improve digital quality. Sharing that knowledge, as with any successful development or QA process, also helps yield success for the business.

How can crowdsourced software testing be integrated with the software development process?

Crowdsourced software testing solutions and strategies can fit throughout the software development life cycle. Whether you need crowdtesting or consultation in the early stages of development, later as the product reaches viability, or later on production systems, a crowdtesting provider can typically lend an assist.

Let’s look at some examples of crowdsourced testing solutions and services at each stage of the SDLC.

Early in the development life cycle:

• in-sprint testing that enables rapid crowdtester feedback on new features as developers code them

• automated functional testing from crowdtesting experts to validate functionality quickly

• accessibility testing and expertise that helps organizations facilitate inclusive design throughout the SDLC

• AI artifact collection to gather the building blocks algorithms need to work effectively — and without bias

Later in the development life cycle:

• integrated functional testing to improve test coverage with a blend of coordinated manual and automated testing

• customer journey testing to validate that customers’ physical and digital journeys with a product, such as buy online, pick up in store (BOPIS), work as expected

• localization testing to ensure that the product will resonate — and not alienate — real in-market users

Digital products in production:

• functional testing to cover any testing gaps or backlog items and ensure proper device/OS combination coverage

• user experience testing, either augmenting internal UX efforts or full-fledged studies, to learn how users perceive your products, including where they experience friction

• AI testing, including validating generative AI results, to ensure algorithms interpret data correctly and deliver proper output

• payment testing to validate that payments process as expected, an oft-complex challenge that can only be tested in production

What tools and technologies are available for managing and conducting crowdsourced software testing?

Organizations that plan to implement a crowdsourced software testing program internally face a variety of challenges beyond recruiting testers. Just to maintain crowdsourced testing, they must develop or provision and maintain a platform that enables users to securely sign up, communicate back and forth, receive defect information and files, compensate testers for their work, enable tester ratings, and filter tester results by their devices, locations and more. It’s a daunting task.

When you work with a crowdsourced testing provider, that partner handles all of the infrastructure challenges associated with sourcing these results. Crowdtesting partners offer a variety of digital quality solutions to address specific areas of need. However, these solutions typically integrate with the organization’s bug-tracking systems to allow for easy validation, triage and remediation. Many solution delivery experts can also work within the organization’s preferred collaboration platform to enable a direct source of communication.

Applause currently offers these crowdtesting solutions and platform capabilities to help organizations achieve digital quality goals.

Manual Functional Testing. Combines exploratory and structured testing to enable organizations to improve digital quality by testing in any country, in any language and on any region-specific device.

Automated Functional Testing. Utilizing instant access to deep domain expertise, this crowdsourced automated testing solution goes from implementation to running in six weeks to enable flexibility, scalability and speed.

Integrated Functional Testing. Make use of manual and automated functional testing in tandem to achieve a holistic, real-world view of your digital products.

In-Sprint Testing. Validate new features mid-sprint almost immediately with a streamlined process that reduces strains on internal resources.

User Experience Testing. Source participants that match customer profiles to run through one of multiple study types that enable you to make informed decisions about your products.

Customer Journey Testing. Receive real-world customer insights in your required locations with testers that match your defined profiles to get a holistic view of your brand’s experience across all channels.

Payment Testing. Test with real, live payment instruments to achieve global, omnichannel coverage and instill confidence that all customers, regardless of payment method or location, can transact successfully.

AI Training and Testing. Train AI algorithms with high-quality, community-sourced data that helps prevent bias, then validate that the results are human-like, accurate, relevant, and not harmful.

Voice Testing. Source diverse voice utterances at scale to build and validate voice experiences for humans of all languages and accents.

Accessibility Testing. A full suite of testing and training services to identify accessibility and usability weaknesses and integrate best practices in accordance with established guidelines.

Bring Your Own Testers. Leverage internal testing teams with a fully managed service that reduces overhead and seamlessly integrates with crowdsourced testing results.

Test Case Management. Maximize testing capabilities with a holistic platform feature that helps manage and execute a testing suite with ease.

How can crowdsourced software testing ensure the quality and reliability of software products?

Software testing helps ensure quality and reliability. The more creative and comprehensive the organization can be with its software testing efforts, the better it can account for real-world use of digital products.

That’s where crowdtesting comes in. While software testers are incredibly skilled at finding defects, there are some issues that escape through to production environments due to insufficient test time or resources, lengthy bug backlogs or lack of relevant expertise or availability when it comes to specific geographical, device or usability needs.

There will always be a limitation to what you can test internally, even with the best testers on the planet. Crowdsourced software testing providers like Applause offer a window to real external digital experts in the locations and with the devices you need to close the gap between a “good” digital product and an “exceptional” one.

Let’s start a conversation today about your digital quality goals and how a partnership with Applause can help you achieve them.