Mobile Testing Pyramid

When most people start with testing for mobile, they tend to start with getting a few real devices and perform exploratory testing manually. Over time, the number of devices will grow, because actual customer feedback comes in describing that their app does not work properly in certain situations on specific devices. Does this sound familiar?

Then is article is for you.

In order to reach fast feedback and lower amounts of manual testing, better usage of available tools is necessary.

What is Mobile Test Pyramid

The mobile test pyramid has 3 levels:

• Real devices (real-life conditions, visual and usability testing)
• Mobile simulators & emulators
• Desktop browsers (using mobile simulation)

Identical to the generic test pyramid, the broader the layer in the pyramid, the more tests you should have which cover a wider range of complexity. (Guess you are familiar with the generic testing pyramid)

Let’s talk about some of the pros and cons of items in the test pyramid

Testing using web browsers on Desktop for Hybrid Apps

• Functional system testing (isolated browser tests performing full functional validations)
• Responsive design (resizing browsers and toggling user agents)
• Cross-browser testing (use equivalent desktop browsers)
• Overall visual layout (no extensive visual checks because the rendering is different than devices)

Pros

• Fast execution: Matter of milliseconds to launch a browser, also headless execution is possible.
• Scalable: Easily set up 10+ browser instances per machine.
• Cross-platform: Ability to use browsers on different operating systems.

Cons

• The mobile simulation uses the desktop browser engine: Mobile simulation in desktop browsers is still using the desktop browser.
• No native integration: No native keyboards, incoming calls, etc.
• Just not a device… Incredibly fast, but still not a real device.

Mobile simulators/emulators

• Functional end-user flows (click paths throughout the application)
• Native API integration (GPS injection, file attachments, incoming calls etc.)
• Visuals (vanilla only) (use equivalent desktop browsers)
• Overall visual layout (emulators are limited to vanilla versions)
• Touch interactions (touch interactions such as swipe and tap comes closer to the user experience of a device than browser emulation)

Pros

• Easy to set up: Simulators/emulators are easy to set up, just download, install, and run.
• Scalable: Virtualization means scalable and also running in parallel on one machine.
• Native API integration: Ability to test native APIs such as incoming calls and GPS injection.
• Simulators or Intel-based emulators are fast: Simulators are fast because they only have to simulate the software part.
• Debugging possibilities: Easy to debug simulator/emulators, already hooked up to the machine to access logs.

Cons

• Vanilla versions only: Manufacturer’s skins are available, but the device behaviour is still based on what comes in stock.
• No real resource usage: CPU/memory usage of the machine in case of simulators. Emulators try to simulate the hardware.
• No real interoperability: Connectivity with NFC, Bluetooth, network connections.
• Slow ARM-based emulators: Emulators based on the ARM architecture are slow, which is the main architecture for Android devices.
• Inaccurate color display in light/dark: Contrast/brightness inaccurate in light/dark environment.

Real Devices

• Usability (validating usability such as actual click areas, touch actions and voice-over)
• Performance (CPU/memory usage, battery, network strengths)
• Native API integration (Interruption like incoming calls, push notifications, resource fighting like camera, GPS, NFC, Bluetooth
• Visuals (focus on devices that are not available as simulators/emulators)
• Manufacturer’s secret sauce , e.g. Samsung’s TouchWiz and built-in browsers

Pros

• Native APIs in real conditions: Ability to test native APIs not only with injections for automation but also actual NFC touch for example.
• Can be faster than emulators: Some real devices are just faster than emulators due to the simulation of hardware, especially compared to the ARM-based emulators.
• Just the real thing… Actual network conditions, battery/CPU/memory usage, manufacturer’s secret sauce on top of the OS.

Cons

• Costs: Real devices come with a price, usually you pay per device/cradle.
• New device means procurement: A new device is usually not available on the fly, even with cloud solutions. E.g. when the new iPhone comes out, it’s not available immediately to procure. In the meantime, the iOS simulator would already be available.
• Development iOS build required for automation: iOS apps need to be signed with Development Distribution Certificate and Provisioning Profile for automation.