Enterprise Pain Points and Best Practices for Mobile Software Testing

Mobile App Testing Trends

Esther Shein

Application testing has become increasingly important with the growing reliance on mobile devices. We recently published an interview with Jean Ann Harrison, Systems/Software QA Engineer with Thales Group/Avionics Division who shared the need for a systems approach to mobile app testing. Here, Frank Moyer, CTO at Kobiton, which makes an enterprise mobile testing software platform, discusses what organizations are doing right – and wrong -- and what some of their mobile testing challenges are.

Enterprise Mobility Exchange: What are some of the top enterprise testing trends right now? 

Frank Moyer: With the rapid adoption of Continuous Integration and Continuous Deployment over the past five years, increasing pressure is being placed on testing. Testers today have to execute more tests more often, with the same team. A common theme amongst recent testing trends is "helping testers do more with less.” For many, the first step is integrating test automation into their DevOps pipeline. This provides a baseline to test their application mobile experience. 

Once test automation is in place, what organizations quickly realize is that those automation scripts don’t maintain themselves. Applications change, test environments are not stable, test automation scripts are brittle, and test execution takes longer than the time allotted. Products are emerging to address these automation challenges, many of them leveraging another technology trend: artificial intelligence.  

Here are some examples. Appsurify prioritizes tests based on code modifications and commonly-used user flows. Testim allows your automation tests to be self-healing, reducing test flakiness. And, finally, Mabl is working towards eliminating the need for writing automation scripts altogether. I expect a sea change in the testing landscape over the next few years as these products start to gain traction.

EME: What are some of the pain points you hear from organizations? 

FM: Within the organization, Kobiton partners with the teams responsible for mobile technology and mobile experience. Our introduction is typically either while they are formulating their mobile test strategy or when they are responding to a production crisis. In both situations, the pain is clear. They want to test the mobile experience on real devices but don’t want to buy and maintain the number of devices necessary to achieve their required market coverage. In the US market alone, it takes 58 device types to achieve 80 percent device coverage! 

And this leads to an important point. With mobile, our customers care less about test coverage and more about user coverage. They want to cover the most commonly-used flows through their application on the most popular devices. We are helping them with two pains: maintaining real devices and prioritizing the devices to test against. 

EME: What common app testing mistakes do software developers make? 

FM: The most common mistake we see is not testing on real devices. The only way to see the user’s mobile experience with an app is to test it on real devices. What usually happens is an organization starts testing on simulators. They experience a production issue that would have been caught by testing on real devices. Soon thereafter they fill out our “Contact Us” form. 

EME: What are some stats on app abandonment and frequency of testing? 

FM: Kobiton conducted a survey earlier this year and learned that: 

--87 percent of mobile testers are testing at least weekly 

--86 percent of respondents say testing real mobile devices is “very important” or “critical”

[Reporter’s note: 22 percent of users abandon an app after one use, compared to 25 percent in 2015, according to Localytics’ latest annual app user retention study. The study also found that app retention has seen noticeable improvement: 38 percent of users launch an app 11 or more times, compared to 37 percent in 2017.] 

EME: How is testing a mobile app different from testing other types of apps? 

FM: Testing mobile apps is significantly different than web testing for a two main reasons. 

First, apps behave differently on different device types; what works fine on a Galaxy S5 may crash on a Nexus 7. Last month, Kobiton performed tests against 50 apps from the Google Play store across 20 device types. While none of the apps crashed on all 20, 24 percent crashed on at least one device. 

Second, mobile apps run on devices under uncontrollable conditions. Location, carrier, concurrent apps, available device memory, and incoming calls are examples of factors that could cause an app to crash. When organizations understand this, they begin to appreciate the resource requirements to thoroughly test a mobile application. 

EME: Who is ultimately responsible for a good customer experience? 

FM: For large organizations, this falls on the shoulders of the chief experience officer or VP of customer experience. At smaller organizations, the chief marketing officer or chief executive officer are intimately involved with the customer experience, especially when app store ratings are at stake. Usually, there is a product owner or someone in a similar role taking responsibility for the application mobile experience.