Mobile Compatibility Testing: When and How

Challenge:

With more and more popular smartphones and multiple mobile network operators, many more smartphones with different demands, brands and producers have been published. How to make your application work as desired on these various mobile devices will be a big challenge. Mobile compatibility testing is indispensable. Meanwhile, with budget control and a tight timeline, it is impossible to perform all the testing across all the devices. A simulator can be a good replacement for the physical device. But is it adequate to perform testing only on a simulator? What are the pros & cons of the simulator vs. real device testing? Do we have a strategy for choosing a simulator or a real device for testing? These will be discussed in the following article.

When to Perform Compatibility Testing

Compatibility testing is not the same as functional testing. Running functional testing against a lot of mobile devices is not advisable. Instead, run the compatibility testing after the functional testing is completed on a single device, or at least after part of functional testing is finished. The test process should cover install/uninstall, functionality traversal, and UI testing.

What Areas Should be Covered?

1. Device/Operation Compatibility

There are some popular operating systems like Palm OS, Symbian, Android, IOS, BlackBerry OS 6.0, Windows Phone 8 etc. You will need to run install testing, uninstall testing, traversal testing, stability testing, UI testing, fluency testing and resolution compatibility testing etc. to make sure your app works well on these operating systems. Different versions of the same OS should be covered, like Android 1.0-4.0.

2. Software Compatibility

Run your app with the popular apps, like anti-virus software, popular social software, general chat software, general browsers, GPS application etc.

For web apps and hybrid apps, the general browsers should be covered in the compatibility testing.

Also, your app needs to be compatible with other apps with similar functionality. For example, run QQ music, Kugou music, WangyiYun music on the same device to test QQ music’s compatibility.

3. Hardware Compatibility

There are various smartphones on the market. The different models of the same brand should be covered in the compatibility testing. Also, mobile phones of different brands with the same operating system should be tested.

4. Data Compatibility

For the information system, the popular database: MySQL, Oracle, SQL Server should be covered when doing the compatibility testing. Also, data exchange with other applications is necessary to be tested for the information system. The data can be words with the format, forms, pictures etc. At last, data from former versions should be inherited and available to the current application.

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

Pros/Cons of Simulator vs. Real Device Testing:

Pros:

1. Simulators are easy to use and can catch the most common bugs. They are also less expensive than physical mobile devices.
2. When the timeline is tight, or there is budget control, purchasing the required mobile devices may be not possible. Therefore it might be necessary to use the simulator in these circumstances for testing.
3. Simulators are, in most cases, open and free software which can be very easily downloaded from the internet and used immediately for testing.

Cons:

1. Testing on simulators can be risky because the users are running apps on real devices, not simulators.We need to consider real devices testing.
2. Real devices are good for testing performance. The numbers are clear and reliable when running testing on real devices. Simulators always run slower than the real device.
3. For validations of battery scenarios, incoming interrupts, and color displays, the real device performs well. Simulators cannot properly simulate these issues.
4. Real devices are better for testing your application under particular conditions, such as a weak network. You can test how your app responds if the network is dropping in and out. You can even test your app on the train or on the street. These are the real scenarios.

The Strategy of Choosing Simulator or Real Device Testing:

As mentioned above, choosing between simulators and real devices for testing shouldn’t be an all or nothing decision. We should focus on the right balance of simulator and device-based testing.

Usually, we should decide whether to run mobile app tests on a simulator or a real device on a case-by-case basis. Budget, staffing, project timelines, and other factors should be considered. For example, you need to track device popularity in the market and also consider the project requirements.

It is advised that you perform testing on the simulator for the initial stage test, then move to the real device for performance, interoperability, network feasibility and regression testing. Also, performing testing on both physical devices and simulators is necessary when the changes involving both the user interface and mobile app architecture.

A New Way: Cloud Testing

With the exception of traditional in-house testing with physical devices and simulators, cloud testing is becoming more and more popular in recent years. It provides a wider variety of devices, almost covering all the popular and latest mobile devices in the market. The user can rent the specific models needed to conduct testing. It supports the typical gestures: rotate, scroll, swipe, tap, tap and hold naturally etc.

The cloud providers also provide specific testing service, like compatibility testing, performance testing. The user just needs to select the compatibility testing service, then upload the package and choose the models. Generally, cloud providers support free use of random devices. Or the user can choose specific models and pay for them. The test can be finished in hours. The test report can be viewed online or downloaded locally.

Except for the sufficient physical devices, the cloud testing is always cost effective and ready-to-use. For specific testing services, it also makes the user’s testing faster and easier. The user just needs to upload the package and wait. Meanwhile, The user may lose autonomy and face a security risk. There is also no universal standard. All cloud providers have their own process and mechanism of architecture, cost, integration, management etc. The compatibility testing service provided by different cloud providers is assigned to different test flows and test points.

When performing testing on a cloud device, the dependency on the internet is a big problem. The user has to ensure an adequate bandwidth to make a smooth connection between PC and cloud.

Conclusion:

Your compatibility testing strategy should be case-based and flexible. The methods/tools you choose should be on a case-by-case basis. Meanwhile, it is also positive to change your strategy as per the functional testing situation.

The combination of real device testing and simulators is a perfect way for in-house testing. It provides adequate devices and ensures security and flexibility.

The use of cloud devices is easy and cost-effective. Meanwhile, it always provides the latest devices. Without the in-house testing constraint, the cloud can be a good replacement for real devices and simulators.

Due to lack of standardization, inflexibility and security concerns, the specific compatibility testing service is not a top option. Of course, it is still applicable for an urgent project or situation with insufficient resources.