Mobile Software Test Automation: What are my options?

Before we get down to the meat and potatoes of why I started writing this blog in the first place, I want to take some time to explore the different frameworks that are out there.  As with anything, there are strengths and weaknesses to each framework.  And how you determine which will work best for you, will depend on your specific needs.  With my current employer, I spent a month or two weighing my options before making my decision.  In this industry, change is about the only given.  I hope that you won't have to spend the same amount of time researching as I did.  Dive in, learn a little, and if it doesn't work well for you, try something else.

What product are you developing test automation for?

This may be a dumb question, because if you are reading this, you likely already know the answer.  But it is important to keep this in mind when choosing a framework. Are you developing test automation for Android, iOS, or maybe a cross-platform app?  Is the app native?  Or is it a web app with a native wrapper, or maybe even a hybrid app?  If you don't already know the answers to these questions, ask someone in your organization.  

Are you comfortable with a certain language?

You may have some prior experience as a developer, or you may have no prior coding experience.  This will certainly play a part in determining which framework will suit you best.  Keep in mind, if you have no prior coding experience, choosing a framework that utilizes a common language within your organization will make it more likely that you could get a hand from a colleague, if needed.  

Decisions, Decisions

When I started writing this post, I had about a half dozen frameworks to talk about.  But when it came down to it, there were really only two worth talking about.  There are some great Android specific frameworks out there like Robotium, Espresso, and UIAutomator. There are also some great iOS specific frameworks like Frank and UIAutomation, but those are specific to a particular platform.  There are really only two names in the game right now that are cross-platform, so those are the two I will be comparing.  

Calabash and Appium

The image above helped me get a better understanding of all the pieces to the automation puzzle, and brought some clarity to the differences between these two frameworks, so i posted it for reference.

Calabash is a cross-platform test automation framework for Android and iOS applications.  It is behavior driven, and can accommodate native, hybrid, or web apps.  Tests are described in Gherkin/Cucumber, and step definitions are defined using Ruby.

Appium is also a cross-platform test automation framework for both Android and iOS applications.  Appium can also accommodate native, hybrid, or web apps.  Tests are described in Gherkin/Cucumber, and it uses JSONWireProtocol to interact with Android and iOS apps using a Selenium WebDriver.  This means that virtually any programming language can be used to code step definitions, such as Ruby, Java, JavaScript, PHP, C#, and Python.  

Both frameworks use a client/server architecture and utilize behavior driven steps written in Gherkin, providing step definitions that are reusable, thus reducing code.  On the surface, they seem very similar.  I use both of these frameworks on a daily basis, and ultimately, you can't go wrong.  But there are some nuances of each I want to discuss, which may help you make up your mind.

Calabash is a great framework, but it is command line heavy.  This may not be an issue for most, but for some, it could pose a challenge.  The interface is actually an interactive ruby shell (irb), and querying objects presents a nice, CSS-selector style view.  Calabash locks you into using the Ruby language.  It is very simple, and mostly needed if you are doing custom step definitions (there is a large library of generic definitions that may work great for your app, and would require much less Ruby knowledge).  For iOS apps, it also requires that you inject calabash.framework and CFNetwork.framework into the project, which are private API's that will get your app rejected from the app store if submitted with those included.  This means that you must duplicate your project and are in essence, running tests against a build that is not exactly the same as the store version.  This may be a deal-breaker for some, but most development shops already do this to a degree with their DEV and QA builds, so it may not be that big of an issue for you.  Now onto the good stuff...Calabash is a breeze to set up, decent documentation from Xamarin, and has a very active community to answer questions.  Personally, I also think the touch events are much more genuine, replicating a more authentic and reliable test.  

Appium has a very user-friendly interface for creating tests.  They also recently released an update to their app that allows test recording, however, this tool is not all its cracked up to be.  The tool only returns code using xpath instead of named objects, which is great for a beginner to learn, but will create very fragile tests if used in a project.  The initial setup is very difficult and time consuming, even for someone with moderate experience.  This, combined with poor documentation, can spell disaster for someone first starting out.  On the plus side, Appium's framework doesn't require you to inject anything into the iOS or Android project, so that you can test the exact same build that will be released to customers, which can be a big plus.  Appium also allows you to write tests using a plethora of languages you may already be familiar with, and is almost identical to Selenium, so anyone with prior web automation experience will likely have a much easier time picking it up.  

Wrapping up

Again, I can't say it enough...Both frameworks are great.  But one may suit you better than the other.  Personally, I prefer Calabash for the reliability of the touch events as compared to Appium.  Also, the irb shell is much easier to navigate on the command line than using a clunky UI, but that is mostly preference.  As I continue this series, I will first go through setup, cross-platform coding, and then integration into a CI process using Calabash.  Once complete, I will start writing about Appium, and maybe some other frameworks that pop up in the meantime.  See you next week!