Any software product, be it a web, mobile or desktop application, is under suspicion unless it proves the features are working as expected. In short, software is never bug free. Depending on the software product, fixing bugs in production is not easy and, in most cases, it’s expensive.
Before the Release of an App
Before a mobile development team ships a native app to the app store, an intensive testing phase must happen to minimize the need for any hotfixes. Distributing the app to beta testers or external testing providers to gain early feedback is another approach to take. However, we all know that the real nasty bugs happen in the wild on customers’ real phones and often during scenarios that only exist in the real world.
The Mobile Bug Matrix
But how to decide which mobile bug is worth doing a hotfix? Over the past few years, I have seen many teams dealing with exactly this question. Some teams tried to fix every bug in production but ended up doing fixes for the released app and stopped doing feature development. Therefore, teams need to find the right balance between doing a hotfix and waiting until the next upcoming release.
To make the decision of doing a hotfix easier, I created the mobile bug matrix. The mobile bug matrix has 2 axes. The x-axis is the criticality/severity for the user, and the y-axis stands for the criticality/severity for the business.
This results in three options. I’ve included an example bug along with each scenario:
- Hotfix: This area has the highest impact to the business and the user. If a bug was found on the live app and fits to this area, the team must fix it as soon as possible.
Example Bug: The login of the app is not working anymore. - Next Release: Bugs in this area either affect more the business side or the user side. The issue is not critical enough to perform a hotfix, but the team must fix it in the next release.
Example Bug: UI elements of the app are in the wrong place. - Upcoming Sprints: These problems need to be fixed but there is usually no pressure from either the business or user. These kinds of bugs can be added to the backlog and can be planned in upcoming sprints.
Example Bug: Wrong translations or typos.
Raise Questions For Clarification
When talking to different stakeholders of the app, there will be different opinions on the criticality or severity of the found bug and in which area the bug fits. In order to prevent exhausting discussions, it might help if the team raises the following questions to make a decision easier to reach.
- How many users are affected by the bug?
- Are we losing activity on our app?
- Is the company losing trust?
- Do we have a legal issue with this bug?
- Are we losing money because of the bug?
- Is the problem reproducible on the supported devices and operating system versions?
- Which sections are affected?
- Can we switch off the feature from the backend?
Once a team has decided if the problem is worth performing a hotfix on, the existing software development approach must be used. Find the root cause for the issue, fix the issue and cover it with some automated checks. Once development is done, a software tester or beta testers must verify that the bug has been fixed. Furthermore, regression testing has to be performed to check that the fix has not introduced additional problems. Last but not least, the whole test automation suite must be executed, and the results must be green.
Find a Workflow for the Team
In summary, the mobile bug matrix is one approach to make a decision on whether or not a bug gets fixed immediately. Every team must define a workflow to handle this situation of critical or non-critical bugs in the production system. It’s important to find the right balance between a hotfix and the upcoming release because fixing a mobile bug isn’t usually fast and cheap. Next to the development and testing costs, there are also app review and submission waiting times for some app stores until the fix is live for the customers, and it might be the case that the next planned app release is just around the corner.
4 min