Testing on the Right Devices

There were more than two billion monthly active Android devices registered in 2017, according to Google. The OpenSignal Android fragmentation report from 2015 found there were more than 24,000 different Android devices on the market at the time. Considering this report is almost three years old, there are even more device permutations available today.

Mobile device fragmentation is not an Android specific challenge anymore. Checking the current iOS devices on the market, fragmentation is also valid for iOS teams.

One of the challenges each mobile development team has to solve is identifying the right devices to use for development and testing because testing the product on all devices is just not possible.

But where to start?

Getting to know the right mobile test device isn’t actually that difficult. The first step each company should do is to check the current statistics of the operating system distribution of the biggest players on the market. Apple provides the latest data about the usage of the iOS operating system here. And Google provides similar data for Android here.

Seeing that iOS 11 is currently installed on more than 60% of all iOS devices and Android 6/ 7 together almost 60% is already useful information for the device selection.

While this is a good starting point, it’s careless if a company is selecting the test devices based on those numbers. Here is an example why this might be a bad idea.

Know Your Target Group

Imagine a mobile app is designed for children. Parents may buy cheaper phones or even give their old phones to their kids in order to use the mobile app. Those phones are very likely to have outdated hardware and will most likely not get the latest operating system updates by the biggest vendors. That means that the above numbers provided by Apple and Google will not apply to the target group and the company may have a problem, because they selected the wrong test devices.

To get more meaningful insights, a company can start with the following.

If a company already has a digital product available on the market (like a website) but not a mobile app, the usage data of the web product is a perfect starting point. Checking the usage data of the web product may give the team the first indication of the used devices. When the mobile app is launched, it is very likely that the same devices will be used by the customers as well.

Creating Personas Help To Identify The Right Devices

If a company want to launch its first ever mobile app and has no other usage data, the selection process is a bit more complicated. As a first step, the company must define a possible target group with personas. With the help of personas, more precise data can be collected with the help of analytics companies who are able to map the defined personas to test devices via real users.

Another source for selecting test devices are statistic websites like StatCounter, where a company can select the target markets or countries where the app will be available to get an idea of the most used devices.

If a company has limited time or money to create personas and to get insights from market researchers, there is another more pragmatic approach they can take. Depending on the target country, it’s worth checking the best-selling mobile devices on the biggest e-commerce website for the particular country. This approach is really simple but may give a company a first idea of the possible test devices. However, with this approach the company may target the wrong audience as explained in the example with the kids app.

Why Working with ‘Hero Devices’ Is Necessary

Depending on the target group and the gathered data, there might be hundreds of possible test devices with different hardware and software combinations. From my previous experience, I recommend selecting the top ten devices (Hero Devices) within the target group. Optimizing the mobile product for those devices will cover, in most cases, the majority of users. If the app has entered the market and there is the need to support more devices, the company can focus on them later by putting the devices into device groups.

How to proceed with finding the right testing devices

As you can see, selecting the test devices for the product isn’t too difficult. If a company has the target customer defined, there are different sources available to get more information about test devices. As a starting point, a company should focus on the top ten devices within the target group to downsize the development and testing effort. If there is the need to optimize for more devices, the test devices can be grouped in different categories with different priorities to focus the development and testing phases.

How to group and prioritize the different test devices will be shown in the next post: How to Group and Prioritize Mobile Test Devices.