iOS testing frameworks and Appurify

Recently, I had a chance to experience some parts in the world of iOS testing. This post is my opinion on various testing frameworks, and the promising, powerful testing and debugging platform - Appurify.

- KIF: tests are written in Objective-C and divided into three "levels": Test Controller, Test Scenarios and Test Steps. Nice structure -- but we will have to create more files. KIF aims to directly manipulate the app and attempts to imitate actual user input. The limitation for that is KIF can not handle system views such as in-app purchase popups, which are outside of the app's scope.

- Subliminal: tests are based on UIAutomation while being written entirely in Objective-C. It was created to solve the problems KIF left - handling system scope views. The tests' structure is similar to the default Apple's unit test's - SenTest - so it is really solid and simple. However, Subliminal is not a good choice for BDD (Behavior Driven Development) since its tests and output may be difficult to read for non-technical customers. Being built on top of UIAutomation, Subliminal does NOT use private APIs so it delivers a proper and safe way to simulate user's interaction. It also captures a screenshot of the device at the moment the test fails. You will find this feature really useful when the test suite gets larger and you do not want to keep your eyes glued to the device's screen all the time it is running.

- Calabash-iOS: it supports Cucumber, hence BDD, and test steps are written in Ruby (C# and JVM will be supported in the future). It has a really nice feature, a console which can be used to quickly inspect the view hierarchy. To interact with the app, there are predefined steps such as touch, swipe, scroll, enter text, etc. Cucumber makes Calabash a truly good choice to do BDD. Both native and hybrid apps can be tested using Calabash (the web view should contain an "<h1>" element with the text "Hello" - from the homepage). Just as KIF, Calabash-iOS uses private APIs but it can handle touch gestures on system scope views. It is also worth noting that from branch 0.10.x, Calabash-iOS will be supplied as a "plugin" to Frank.

- Appurify (currently in private beta) is an awesome enterprise-class platform made for testing, debugging, and optimizing performance of mobile apps. Appurify supports many testing frameworks including Appium, UIAutomation, Calabash, OCUnit, KIF. Subliminal is not supported (yet) although it is on the list on their FAQ page. There is also a simple test called "Unleash The Monkey" which just taps around randomly for a while. This is helpful when you need a simple way to check for obscure bugs.

You can submit tests using the website or scripts. By using scripts, you can integrate them with Xcode to have the tests submitted and run after each build. Tests submitted to Appurify are run on a large pool of real devices that are configurable. You can choose among various iPad and iPhone models then config the available memory, network status (Wi-Fi or cellular, rich or poor signal strength) and device's location. Those devices are non-jailbroken. The test results include logs, device stats, instrument trace, screenshots, and even a video. A lot of information.

One thing I loved about Appurify is that it used to allow us to interact with the devices through their website but this feature did go away with the recent launch of the new UI.

Appurify also comes with an SDK which is currently only supported on jailbroken devices. The SDK includes the web inspector, network capturing, device stat viewer (memory consumption, battery capacity, thread count, etc.), and crash-the-app-now features. You can also record your interaction and upload it to their website to have it converted to UIAutomation tests. The web inspector allows us to inspect in-app web views using the desktop version of Safari or even Chrome.

Despite some existing issues, Appurify is still in beta so it will surely be improved over time.