How Can You Run More Mobile A/B Testing Experiments?

Imagine you and your team just a released a redesign of the login section of your software product. Shortly after the release you notice that the numbers of newly registered users dropped almost to zero. But why? Is it because the change of the register button to a register link? Or is it the new naming of the register element?

Maybe this kind of change should have been tested before with an A/B test.

What is A/B Testing?

A/B testing in the context of digital software products gives a team the power to test and compare ideas like in a science experiment with real users. Basically everything can be tested: the color of buttons, new text for navigation elements or a complete different version of the product. A/B testing will help teams understand the impact of the changes on a subset of real users to get real insights.

Every A/B test starts with the creation of a hypothesis. A strong hypothesis is needed to describe the A/B test upfront and to explain the expected outcome. You can use the pattern:

Changing__________ from__________to__________will__________.

An example:

"Changing the register button from button to link will increase the sign-up rate."

This is just a theoretical assumption but it will help you to solve the problem.

Once the hypothesis is created, it’s important to define a test user group. Group A is the control group, which still gets the existing product and group B gets the modified version. If one group is performing better according to the hypothesis, then it should be applied.

How Can You Run More A/B Testing Experiments

So how can you start with A/B testing in your mobile project? First, you need to find the A/B testing vendor you want to use in your apps.

All vendors provide an A/B testing SDK (Software Development Kit) that needs to be added to your application. The integration is a one-time task and should not take too much time.

Once the SDK is installed, you can start to create an A/B test on the mobile app using the dashboard of the A/B testing vendor. Most of them offer a web based dashboard, that can be used to configure the A/B test, as well as select the users for the control and testing groups. From the dashboard, you can see the progress of the A/B test as well as the results once the test has been finished.

However, there are two things you should keep in mind when adding an A/B testing SDK.

  1. Users need to update the mobile app with the included SDK in order to be part of an A/B test. If users don’t have the right version installed, make sure to target the test group based on the app version.
  2. Adding the SDK will increase the size of the mobile app. Make sure to update the app to the latest version in case the user doesn’t have enough memory left on the device.

Benefits of A/B Testing

By adding an A/B testing SDK to your mobile app, you gain a number of benefits that are worth increasing the app size as well as the one-time integration effort. The following list includes an excerpt of the benefits you gain:

  • You can run multiple A/B tests at the same time
  • You can extend the A/B to an A/B/C../Z test
  • The findings of an A/B test can result in higher conversion rates
  • It will be easier to make decisions based on real customer usage data and statistics
  • You gain more insights about customer behavior
Want to see more like this?
Daniel Knott
Mobile Testing Expert
Reading time: 5 min

How to Assess Testing Resource Allocation

If you can’t measure the value of your efforts, you can’t explain or even justify your testing investment

Using Questionable Datasets to Train AI Could Come With High Costs

As companies look to capitalize on AI development, they must stay mindful of how they source training data — AI algorithms developed from private or non-consensual data may cost businesses in the long run.

Why Payment Testing is a Constant for Media Companies

Simulated transactions and pre-production testing won’t ensure friction-free subscriber payment flows

How Mobile Network Operators Can Leverage e- and iSIMs

We explain what e- and iSIMs are, what they mean for the industry and how MNOs and MVNOs can leverage them to their advantage.

Jumpstart Your Software Testing Education

Testers have a variety of options to upskill and grow professionally

The Future of Generative AI: An Interview with ChatGPT

We ask ChatGPT about where it sees itself in the future, what needs to happen for it to get there and how Applause can help.