Mobile Software Test Automation: Physical devices or simulators?

Should I use a real device, or is the simulator adequate?  That is a good question, because the answer really hinges on what your chief aim is in testing.  Are you looking to set up test automation as part of a CI process, and use this as more of a smoke test?  Are you are looking to create a full test automation regression suite to give yourself more time for exploratory and feature testing?  Maybe a little bit of both?

Pro's of Simulators/Emulators

There are definite advantages of running tests on simulators/emulators.  But this needs to be broken down by platform for clarity.

iOS

iOS simulators are cost effective (read free).  Xcode's Developer Tools comes with a number of simulators on install.  Basically, any device that Apple supports, you have a simulator for.  They are easy to use, and don't require any special certificates or provisioning profiles to work.  They launch fairly fast, and are also very reliable.  

Android

Android emulators have much fewer advantages, but cost may be an advantage for some.  You can also create an emulator using the AVD manager that comes with Android Studio for just about any device configuration there is, which may also be helpful.  There are also a lot of good prebuilt emulators from Genymotion that are reliable, but you have limited options with the free version.  And that pretty much wraps up the advantages for emulators.

Con's of Simulators/Emulators

iOS

One of the con's of the iOS simulator with regards to test automation is that you can't use any of the device extensions, such as the camera, phone, Bluetooth Low Energy, push notifications, etc.  Also, memory management could play a big role, so heavy apps may run fine in the simulator, but could crash on an actual device.

Android

The list of con's for android emulators is quite extensive, but speed is probably the greatest factor for me.  Launching an emulator takes FOR-E-VER.  So if you are holding up a deploy until your smoke test runs, this could be a pain point.  Memory management is a little better than compared to iOS simulators, but not much.

Summary (TL/DR option)

The simulators are great if you are looking mainly for a UI rendering, or for a tool that other teams can use without having to hog devices you have allocated to smoke/regression tests.  But beyond that, there is NO SUBSTITUTE for a real device.  There are definite pain points (specifically with iOS) when getting everything set up to run on a device, but once its working, you shouldn't have any problems.  In the end, the people using your apps don't use it on a simulator, so why would you test on one?  Real Devices are costly, but the confidence gained from valuable tests on real devices, and the speed at which tests run on a real device (mainly android) far outweigh those costs.

Now that we have all the background topics done, next week we will start getting into setting up your local environment, so stay tuned...