• Hulu
  • TV
  • Movies
  • More TV. On more devices.
Search
Hulu Tech Blog

GSAutomation — an open-source iOS test library

April 24th, 2013 by Bao Lei

We all wish testing applications was as much fun as watching Hulu content. Just sit back, watch some Family Guy or The Office, and the app is fully tested. While sometimes this is part of our quality assurance process, most of the time testing is much more mundane. Logging in, logging out, resetting passwords, adding items to your queue, checking watch history, checking search results, rotating the device, making sure you can’t watch Chasing Amy when logged with a 13-year-old’s account, ensuring correct ad logic… And as tedious as this might be to do on a beautiful new iPad with a Retina display, you have to repeat this process on the iPad 3, and iPad 2, and on the iPad 1. And then a new build comes, and we do it all over again.

Test automation is the obvious solution. On the iOS team we’ve traditionally relied on Apple’s UIAutomation, which is a cool framework that lets you simulate user behaviors (such as taps, scrolling, flicks, rotations, string input, etc) and check UI elements in JavaScript. At first, watching buttons get magically tapped by an invisible ghost navigating throughout the app is impressive. Eventually, however, we found ourselves spending more time maintaining the script than the time saved from the automation process. One reason is that the automation script covers a rather small portion of test scenarios, and the other reason is that the automated tests are sometimes fragile, causing false negatives due to a changed application view layering rather than a legitimate bug.

With the goals of making test scripts easier to write and making their performance more reliable and robust, we created the library called GSAutomation. (Wondering where the prefix GS is from? Check out our last year’s blog post on GSFancyText.) It is an extension/wrapper for UIAutomation, which makes iOS app testers’ and test script developers’ lives a lot easier.

First, GSAutomation makes writing scripts parallel real human testers’ behaviors. For example, if you want to tap a button and then check if some label says the right thing, just define a task array like:

task = [
  [Tap, "Button"],
  [Check, "Text I'm expecting", "Text I'm expecting from another label"],
]

And then call performTask(task)

A task array consists of a number of steps. Each step is itself an array starting with the action name, followed by a series of parameters (e.g. for “Tap” it’s the title of a button, for “Check” it’s a list of labels/text views.) The simplicity of this syntax means that it’s no longer a requirement to be a software developer to create a test. Anyone with a text editor and some light education about the syntax can chip in and get their favorite features covered by tests that are running nightly.

But why arrays? Why not just define some helper methods and then make the scripts like

Tap("Button");
Check("label1", "label2");

One reason is of course that in the Cult of Objective-C we think square brackets are more beautiful than parentheses. Another reason is that while interating with the task arrays, we got many instabilities and sharp edges handled with some common logic. So the scripts are more robust against all scenarios. For example, there is always one second delay between individual actions. Also, for tapping or checking results, if the element you are looking for doesn’t exist at the beginning, we patiently wait for some more time since we know it might be either network latency or an old device’s poor CPU.

To make tests even more robust, GSAutomation offers a failure rescue mechanism for some actions. For example, when we test the player within the Hulu Plus app, if our script fails to tap the pause button it might not be because pausing is broken — instead, it could have been that the control bar auto-hid after seconds of inactivity. So we simply tap the screen center and try the pause button again. The step array here will be:

[Tap, "pauseButton", [TapPoint, screenCenter()]]

The last parameter in this step is the rescue action if the original action failed. Note that in this example, the pauseButton doesn’t have a text title, so we refer to it by the image file name — credit for this flexibility goes to UIAutomation (and more fundamentally, UIAccessibility, which determines how UI elements are referred to within UIAutomation).

Other than this task array based workflow, GSAutomation also provides a list of helper methods to work around the need to use Apple’s long method name convention and make basic jobs much simpler. So for example you can call isPad() to check whether it’s iPad; you can call win() to get the main window; you can call log(“some text”) instead of UIALogger.logDebug(“some text”).

Interested and want to give a try? Check it out at http://github.com/hulu/gsautomation

The project includes an example with a simple iOS app project and GSAutomation based test scripts. For full definition of actions, parameters, and ways to reference an UI element, check the README on Github.

Bao Lei is a software developer on the mobile team who works on our iOS platform.

Last comment: Aug 20th 2014 3 Comments
*
*