3 Key Crowdtesting Use Cases
Crowdtesting is a scalable and cost-efficient way to guide product decisions and improve quality. It enables companies to expand into new markets and gain deep insight into specific industry use cases, and it can deliver on customer demands for faster testing, which means faster time to market.
What is crowdtesting?
Crowdtesting — alternately called in-the-wild testing, community-based testing and real-world testing — leverages custom and vetted testing teams sourced from a community. Applause customizes teams of highly vetted digital experts from around the world, ready to stand up testing at a moment’s notice. They can test in any language, on any device and operating system. Our crowdsourced testing teams are skilled in the full suite of testing solutions – from manual exploratory testing to accessibility and payments testing. A well-sourced crowdtesting group serves as proxies for your customers.
There are several key high-level benefits of crowdtesting for development organizations:
Faster innovation with fewer bugs: Through rapid testing and feedback, QA teams can better keep up with an increasing pace of development, while improving quality – helping you get to market faster with higher-quality digital assets.
Improved customer experience: Authentic feedback from real users in real-world settings helps you improve app performance and deliver better customer experiences – increasing conversion rates and driving retention.
Reduced testing efforts and cost: Remote, distributed teams help you scale test coverage without the burden of additional overhead costs.
Access to skilled testers with specialized knowledge offers significant business value. I cover a few specific use cases below.
1. Crowdtesting and product launches
Crowdtesting complements in-house QA and extends the organization’s capabilities, without replacing internal resources. This can be particularly important around product launches where time to market matters. Crowdtesting can provide a sense of relief for QA managers — especially those with small teams — by freeing up in-house staff to focus on more strategic tasks. For example, for new product launches, a crowdtesting partner can conduct regression testing with structured test cases to ensure that the new feature(s) didn’t break any existing functionality. In addition, real-world testing can assess new areas of the application that may not have been considered by the internal QA team.
In the majority of cases, crowdtesting is used to:
Fill gaps or offload repetitive, time-consuming tasks from internal teams
Test in locales where the the organization has no presence or ability to test
Test on a diverse array of devices and operating systems practically unavailable to the QA organization
However, there are cases where a crowdtesting team conducts most of the testing for an internal team, as is the case with Applause and Expensify.
An example of complementing an internal QA team is one where the internal team focuses on functional testing for core user flows, while the crowd-based testing team performs exploratory tests to help product teams uncover valuable issues that structured test cases might miss, and to better understand how customers actually behave. This is a critical component when it comes to product launches, as both testing flows can be done simultaneously. A great example of this is our partnership with Uber.
An organization might use a crowdtesting team to evaluate on devices, networks or operating systems the internal team doesn’t have available. Again, testers give real-world feedback on how an app performs before it reaches production. Crowdsourced testing teams can also explore scenarios that can’t easily be replicated in a lab, such as geofencing for gambling apps or streaming media on a roaming network during travel. Using an emulator for geofence or roaming testing is not a valid test for these scenarios; actual in-the-wild testing is the only way to uncover most issues.
No one wants to roll out a new product without validating that it works in the real world, which is only possible through the testing rigor that crowdtesting provides. This in-the-wild testing empowers organizations to quickly scale to support new product launches without hiring additional staff.
2. Leverage the efficiency of crowdtesting for market expansion
Your plans include expansion into a new market. Hopefully, you’ve already built a customer-centric product, having used real-world feedback from a crowdtesting partner, but does your product knowledge in one market automatically extend to another?
Crowdtesting is one of the only ways product organizations can confidently expand into new geographic regions, without having to hire additional staff. There are three key areas that can support your expansion.
Geographic coverage
No matter how advanced your VPN technologies may be, using a VPN to test a digital product in various simulated locations inevitably causes unexpected issues. QA teams relying solely on these tools will always face some uncertainty as they can’t predict how they affect the functionality and interface of their app. Using a VPN can cause a delay in loading certain data like geo-localized ads, for example. This means that there’s no accurate way to test those ads, rendering the testing scenario useless. Other issues that can’t be accurately tested without using people in real-world scenarios include: using data on partner networks around the world; payment testing on specific mobile networks; testing GPS-related features and more. The same goes for geo-fencing testing, which tests that apps work in prescribed areas.
A current online sportsbook vendor requested that Applause source, vet, manage and train testers who live within a local border to register for their online sportsbook, possibly involving in-person registration. The company wanted to ensure its customers could place bets when located 1 and 5 miles within the border, but not when located 1 and 5 miles outside the border. Applause responded to the request and, in addition, sourced new testers for each new launch and for every new locality.
Localization
When you ensure you’ve fully vetted your app in a local market, you improve the UI, which in turn improves customer loyalty and creates a cycle of excellence. Customers who respect and appreciate your brand are more prone to give you feedback. In addition, your users quickly notice when you have made the effort to understand local idiomatic expressions and nuances that may require less literal translation and more conceptual communication. Bottom line - they know you care.
Whether it’s fully understanding cultural norms or space requirements/design in local language, payments functionality, digital accessibility (as each area of the world has different regulations and laws) or ensuring that app stores are optimized so users can quickly find your app using local language, it’s key to use in-market, custom-curated crowdtesters.
Device coverage
Device fragmentation creates a significant hurdle to testing new apps. In its blog from 2019, scientiamoble mentions tracking more than 63,000 device profiles at the time. It also reported that the device profile growth rate was approximately 20% per year. The sheer number of devices and operating systems is enough to overwhelm any QA department. Even if you had unlimited resources to hire full-time employees to grow your device lab, you would be challenged to purchase, organize and test for this incredible variation in devices and OSes. Crowdtesting solves this issue, as it gives you instant, global access to any device and operating system and helps ensure you uncover all unexpected bugs and edge cases.
Using crowdtesting to ensure successful market expansion is key. Through localization - inclusive of thorough geographic testing - QA teams can provide the best customer experience, from design issues to translation, currency display to payment processing.
3. Real-world testing for specific industries
If you’re in the market for a crowdtesting partner, then the partner’s industry experience should be a key focus in your evaluation criteria. Does the prospective partner have deep and broad industry experience? At Applause, we have years of expertise in: retail, media and entertainment, financial services, travel and hospitality, healthcare, and IoT, among others. We also have extensive experience in specific customer use cases that are critical to getting testing right.
Here are two focused examples:
Streaming media
The streaming media sector is a highly competitive space. Providers must ensure seamless content delivery and a high-quality user experience to attract and retain viewers. As test automation is inherently not a good fit for testing streaming content, in-the-wild testing with real people enables companies to leverage manual testing for functionality and UX, and Applause helps streaming media providers around the world test the following industry-specific areas:
Feature usage. Search; rewind/fast forward; playback; pausing; customer profile; favorites; value-added features
Content validation and management. Profile management; personalization and localization of content; images; previews; shows; movies
Multi-channel experiences. Continuity across platforms and devices (mobile, smart TV, gaming systems, OTT etc.)
Live event monitoring. Streaming quality across devices; stress testing; real-time feedback
UX. Seamless navigation; consistent experience; personalization
Localization. Translations/dialect expertise; captions; metadata; cultural norms
Accessibility. Closed captioning; color contrast; UX for people with disabilities
Subscription process. Registration; authentication; renewal; introductory rates and free trials
Ad operations. Ad quality; seasonality; personalization and localization; confirmation of ads, ensuring they are clickable and change in cursor
Applause helped a large media provider expand its streaming capabilities into Latin America by ensuring a localized viewing experience through rapidly sourcing native language speakers in 25+ countries. We sourced testers to meet specific language, device and payment instrument requirements. Testers performed content validation and localization testing to ensure episodes are accurately displayed in each market.
Retail
Not surprisingly, ecommerce is one of the major retail trends of 2022. Whether it’s omnichannel or AR, voice or sustainability, retailers have their carts full when it comes to the ever-increasing demands of customers wanting new, personalized experiences that excite and that flow seamlessly from one channel to another. Applause helps retailers do just that, ensuring the best digital and physical commerce experiences possible. We help retailers test:
Content validation. Personalization; promotions
End-to-end experience testing. From search to purchase to return
Omnichannel strategies. BOPIS; endless aisle; ship to store; ship from store; BORIS
Payment instrument testing. Local testing in global markets
Ecommerce replatforming. Template testing; brand-level conformity; ongoing maintenance
Accessibility testing. Assessments; UX; consulting; training
Voice testing & validation. Voice ordering; UX feedback; payments
Loyalty strategies. Reward redemption; UX
Shake Shack wanted to ensure both app and in-store experiences live up to Shake Shack’s and customers’ high-quality standards. Applause sourced testers to travel to specific locations to stress test the app as well as the operational processes while in the restaurant. Applause provided detailed technical and UX feedback throughout the customer journey.
Read the Shake Shack case study.
Crowdtesting, in-the-wild testing - whatever you call it, its benefits are very clear
Applause helps our clients realize substantial value and return on investment in critical business areas and the ROI for crowdtesting is clear.
- Cost-effective: You can have thousands of people on thousands of devices testing your app or site without employing the testers or buying their devices.
- Speedy: More people at work means faster execution.
- Unbiased: Crowdtesters are not wired to support or defend any part of your app or site, as their different working locations means confirmation bias, groupthink and internal concerns of your company don’t affect their opinion.
- Data-focused: There is diversity of opinion, independence, decentralization, and aggregation in crowdsourced testing.
Learn more about how crowdtesting provides competitive advantages in our ebook, The Essential Guide to Crowdtesting.
