Test Automation Basics – Mobile Apps & Web and Games
If we are considering mobile application ecosystem of android and iOS, we find it dynamic, as both are suffer from software and hardware fragmentation. Especially, it is applicable for Android, however fragmentation also available in iOS ecosystem, as experienced with the launch of iOS 8.
There are many existing iOS applications available with its latest version that made clumsy on updated devices. Moreover, one should find typical issues in the new iPhone 6 and 6 Plus for Apple devices. Users with some options will find an important amount with older devices, especially to purchase new device for getting everything working well.
If we are considering Android, everything you find is different. Application and game developers are getting difficulty as OEMs launch software updates, latest devices, and customizations for their devices to maintain product fully supportable with all possible device alternatives. Creating some application and game work on quality devices is out of the question.
One-stop solution for common problem is professional automated testing software. It enables to create robust and reliable software with difficult technology and difficult competitive pressure. One of the affordable solutions is Automated software testing to this problem. It delivers three business advantages like:
- Increased testing effectiveness
- Faster time to market
- Increased testing efficiency
Test Automation is best For Mobile App Development
Creating robust mobile apps is different from developing embedded and PC software. When developing mobile app, developer have to use tools and practices for that agility.
But doing something manually like testing an application is never agile, and thus, test automation has shown huge growth among developers and application, improving robust and better outcomes. Test automation as the part of the agile development procedure is normal for attaining better compatibility with devices, market and users.
Interestingly, many tools are available, but test automation stands on the first position. A significant value-add is offered by this tool to enable testing to be done in similar way. The next day, the tests will be confirmed and the outcomes will be ready for review.
As we all know that fixing a problem saves huge time and enable developers to finalize products earlier. Importantly, it gathers better quality with limited number of bugs.
Factors to Consider Automating Mobile App Testing
Now, we have to check how test automation can enhance the development process, increase speed up development and add value.
Costs And Assets
One will require time, people, infrastructure, tools and training irrespective of whether you go for manual or automated testing. Test automation will deliver a good ROI, depending on the project and application. Moreover, Automated Software Testing helps to minimize repetitive tests from weeks to hours. It is time saving that translates directly into cost-saving.
Incorporated Testing Cycle with App Development
Increase in efficiency and productivity with automation starts with catching a new mindset. During all development cycles, software tests have to be repeated to make sure the possible application’s quality. Software tests should be repeated when source code is different every time.
The software should be tested on all supported variants of OS for each release and all configuration of hardware. However, manually repeating would be costly and time-consuming.
Tools And Technology Used for Test Automation Frameworks
First of all, you have to choose the most robust and most cross-platform method for getting the most out of efforts and maximizing testing coverage. Moreover, one can use such automated methods for validating needs and to minimize costs through test-case generation.
But, the full automation of large software entities is available with a cost that some companies haven’t been read to pay.
Test Coverage And Reusability for Open Standards Mean
The depth and scope of tests can be increased by automated testing and importantly enhance software quality. Some long and thorough tests can be run automatically often not doable with manual testing.
Well, test cases comes with a full access to an application and test all factors, including data tables, file contents, memory contents, and internal program states to decide whether the product runs according to expectations or not.
With automated tests, one can easily execute many difficult tests when running test to deliver coverage that is not possible with manual testing. Moreover, developers have enough time to develop new automated test cases and integrated more interesting features.
Improve Usefulness And Conclude Sooner
Professional automated testing software is one of the best solutions for common problem, including how to create better and robust solution with ever growing difficult of technology and under massive competitive pressure.
Business results are improved by automated testing in three ways, i.e. greater testing effectiveness, greater testing efficiency, and a shorter time to market.
Various ways to Automate Mobile Testing
There are three ways to automating the testing of mobile applications such as:
Handwritten Test Scripts
Generally, it is one of the preferred options when you have idea about what you are doing and when you have programming-capable resource for doing the test scripts. Different alternatives are also available for test automation tools, frameworks, and integration for both open-source and commercial offerings.
Record-Playback Approach
It is the least error-prone approach as it nothing has written in code, however has limited functionality. Tests can easily and quickly recorded and after that, played repeatedly against different OS versions and device configurations. Such test concentrates on user-driven and user interactions activities.
Automatic Test Exercisers
A great way to smoke-test applications is provided by automatic test exercisers. No particular tests are required instead of focusing on testing user-interface logic like clicking buttons, opening menus, swiping and multi-gesture actions.
The least exact results yielded by automatic test exercisers, however delivering quick feedback on iteration of an application.
User Interface And Functionality
One can identify the success of application and games with users interface and its overall functionality. The two things that include visual appeal and gameplay are the important things to get right and one must ensure that device fragmentation doesn’t break any of these.
Different things in the UI require to be checked such as:
- UI layouts and elements
- Menu structure and functions
- Screen orientation
- Screen resolution
Graphics Performance
Software performance should be consistent across all devices variants among your users and test on many real devices are possible. It is advisable to create tests for determining how well your application and game responds to various levels of usage, including battery usage and performance that consider creating tests that last for hours.
Run load tests for determining whether your game runs effectively under a heavy load for a long time. Such performance test will evaluate like how responsive your game is on real devices.
Usability And User Experience
User experience, testing usability, and navigation flow cannot be completed on a desktop with a keyboard and mouse. So, it is advisable to use real devices instead of forget emulators.
You have to test how usable and entertaining your application is by considering two important things like
- Background events
- User interaction and responsiveness
Multi-User Features
Presently, multi-user support is common in both games and applications. Moreover, testing multi-player capabilities also play an important role and are more challenging needed real users for measure performance.
A game communicating is a typical case with the back-end server. One can also synchronize the back end with devices for better connectivity to get information about the gameplay.
Moreover, one can check test of various scenarios among them could affect on the game’s experience that results into negative feedback and lastly, game being uninstalled by players.
Social Integration
Another important factor is to integrate with social networks. In next step, you have to test this thoroughly with real iOS and Android devices with different operating system versions and different device configurations to measure functionality and ease of utilize.
Security And Liabilities
Some open-source components used by all developers in their applications. Such type of practice is used and recommended as it offloads the code’s development for non-core functionality. But, developers neglect identifying vulnerabilities and licensing restrictions with third-party code.
Breakdown: Android Test Automation Frameworks
Robotium – It is one of the popular Android test automation frameworks, supporting native and hybrid applications. With Robotium, one can easy to write robust and powerful automatic black-box UI tests for android applications.
UIautomator - delivers an efficient ways for testing UIs. An automated functional test cases are created that can be implemented against applications on real Android devices and emulators.
Calabash - If you are looking for framework for Android and iOS then Calabash is a cross-platform test automation framework. The best thing about Calabash is it understands syntax that enables non-technical people to develop and implement automated acceptance tests for apps on both of these mobile platforms.
Appium: Automation of native, hybrid and mobile web apps.
Appium is one of the best mobile test automation frameworks for mobile-web, native, and hybrid apps for iOS and Android. JSONWireProtocol is used by Appium for cooperating with Android and iOS application by using Selenium’s WebDriver.
Written in Node.js, Appium is an HTTP server, developing and handling multiple WebDriver sessions. The test is started by Appium on the device and listens for commands from the main server. It seems same as the Selenium server that gets HTTP requests from Selenium client libraries.
How to Set Up The Environment?
First of all, download our Appiumexample. You can find example in Java, Python, and Ruby. After that, choose clientlibrary according to programming language you will choose. For more information, you can switch to Appium’sdocumentation where all guidelines, help and extra material are available.
MAC OS X AND LINUX
Firstly, you have to make sure that Python 2.7.x or later are installed (Note: It’s preinstalled on Mac OS X):
$ python --version
Now, you have to install Python (Linux users only):
$ sudo apt-get install python2.7
After that, you have to check Python’s “pip” module is already installed or not:
$ pip --version
Install pip if you’re on Mac OS X:
$ curl
https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ sudo python get-pip.py
$ rm get-pip.py
Install pip on Linux (if it is not installed)
$ sudo apt-get install python-pip
In next step, you have to install the Selenium module for Python:
$ sudo pip install selenium
Now, you have to verify that Selenium is installed:
$ pip freeze | grep -i selenium
WINDOWS
You have to make sure that Python 2.7.x or later are installed:
$ python –version
If it’s not run properly then you have to download and run the setup from Python’sdownload center. In order to add Python environment variables, you have to go to “System properties” → “Advanced System Settings” → “Environment Variables” → “System Variables” → “Edit Path,” and after that, insert C:\Python27\;C:\Python27\Scripts\ at the end. It is supposed that you have installed Python in the default location.
You have to ensure to restart the command prompt for inviting new environment variables in to effect.
Now, you have to check if Python’s pip module is installed or not:
$ pip --version
If you don’t have install, then install pip:
$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py > get-pip.py
$ python get-pip.py
$ del get-pip.py
Lastly, install Python’s Selenium module:
$ pip install selenium
Running The First Tests
In that step, you have to download sampletest script and sample application.
Download the Appium sample test script and sample app. The APK for Android, the IPA for iOS and one sample test script are available in the sample package. Furthermore, three samples are
- appium_example_ios.py
- appium_example_android.py (for Android API level 17 and above)
- appium_example_selendroid.py (for Android API level 16 and below)
Step: 1 Make Account And Upload The APK
You have to create an account. A freemium option is also provided by this service and you don’t require a plan for completing these samples by using real devices on a cloud service.
One can have different plans for accessing hundreds of device models. One can have to upload the APK or IPA to Testdroid’s cloud service before proceeding with running the test script through HTTP POST. Now, let’s try using cURL.
For Mac OS X and Linux:
$ curl -s --user username@example.com:password -F myAppFile=@"/absolute/path/to/app/SampleApp-iOS.ipa" http://appium.testdroid.com/upload
FOR WINDOWS:
$ cd C:\path\to\directory\containing\the\application
$ curl -s --user username@example.com:password -F myAppFile=@"SampleApp-iOS.ipa" "http://appium.testdroid.com/upload"
You will get a JSON response with status: 0 and a unique identifier for the uploaded app upon successful upload.
{
“status”: 0,
“sessionId”: “abcdefg-1234-5678-abcd-111122223333”,
“value”: {
“message”: “Uploads Successful”,
“uploadCount”: 1,
“rejectCount”: 0,
“expiresIn”: 1800,
“uploads”: {
“myAppFile“: “abcdefg-1234-5678-abcd-111122223333/SampleApp-iOS.ipa“
},
“rejects”: {}
}
}
Step 2: Set Credentials And Other Parameters
Now, open the test script: Appium_sample_ios.py, in any text editor. You have to set screenshotDir to the path, where you want those screenshots to be kept on your machine. In DesiredCapabilities, you have to set your credentials on testdroid_username and testdroid_password.
After that, you have to set the myAppFile identifier against testdroid_app in Desired Capabilities. It seems like:
desired_capabilities_cloud = {
‘testdroid_username’: testdroid_username,
‘testdroid_password: testdroid_password,
‘testdroid_project: ‘Appium iOS Project 1’,
‘testdroid_target: ‘iOS’,
‘testdroid_description: ‘Appium project description’,
‘testdroid_testrun: ‘Test Run 1’,
‘testdroid_device: testdroid_device,
‘testdroid_app: ‘sample/SampleApp-iOS.ipa’,
‘testdroid_platformName: ’iOS’,
‘testdroid_deviceName: ‘iPhone device’,
‘testdroid_bundleId: ‘com.bitbar.testdroid.SampleApp-iOS’
}
Step 3: Run The Test Script
You have to implement the below given command:
$ python sample-appium_ios.py
The output of console seems like:
Click here for see this image.
Step 4: Get Results From Cloud
You can see screenshots available locally on your machine in the directory that specified in Step 2. Now, you have to log into your cloud service and navigate to the project’s name, as defined in the _project attribute of DesiredCapabilities for accessing following log files:
- Appium’s server log,
- Logcat and instruments log.
Some Advanced Example: By using Appium for mobile game testing with image recognition.
We are using Supercell’s popular mobile game Clash of Clans in this example. It is one of the interesting games, many of you have played it and thus, you should be familiar with its gameplay and look. It is one of the best example of basic clicks through game’s tutorial.
## Example script that tests Clash of Clans tutorial first steps
## Works on different resolutions, both iOS and Android
import unittest
from time import sleep
from AppiumCloudTest import AppiumCloudTest, log
from selenium.common.exceptions import WebDriverException
class ClashOfClansTest(AppiumCloudTest):
def setUp(self):
# AppiumCloudTest takes settings from environment variables
super(ClashOfClansTest, self).setUp()
def test_tutorial(self):
driver = self.get_driver() # Initialize Appium connection to device
sleep(10) # Wait for game to load
# Use this to get detected screen hierarchy
# print self.driver.page_source
# Dismiss the in-app purchases dialog if it shows
okayButton = None
if self.isAndroid():
try:
okayButton = driver.find_element_by_id('button3')
okayButton.click()
sleep(5)
except WebDriverException:
log("There was no in-app purchases dialog")
else: # iOS
self.driver.implicitly_wait(5) # Wait only 5 seconds to find it
try:
okayButton = driver.find_element_by_accessibility_id('Okay')
okayButton.click()
# No need to sleep here
except WebDriverException:
log("There was no in-app purchases dialog")
self.driver.implicitly_wait(30)
# Cancel iOS Game Center login
if self.isIOS():
#print self.driver.page_source
try:
self.driver.implicitly_wait(5)
cancelButton = driver.find_element_by_accessibility_id('Cancel')
log("Cancelling iOS Game Center login…")
cancelButton.click()
sleep(2)
except WebDriverException:
log("The Game Center login was not displayed")
self.driver.implicitly_wait(30)
self.screenshot("welcome-chief")
# Check that a goldmine is on screen
rect = self.find_image("queryimages/tutorial_goldmine.png",
screenshot_match="screenshots/goldmine_match.png")
self.assertIsNotNone(rect,
"There should be a goldmine on screen at beginning of tutorial")
log('Gold mine found at %s %s! Continuing…' % (rect[0], rect[1]))
# Dismiss the bubbles
self.tap_middle()
sleep(2) # second blabla
self.tap_middle()
sleep(2) # Goblin appears
self.tap_middle()
sleep(1)
# Go to shop
# NOTE: tap_image does also assert, fails test if target not recognized
self.tap_image("queryimages/shopbutton.png")
sleep(1)
# Buy cannon
self.screenshot('cannon')
self.tap_image("queryimages/cannon.png")
sleep(2)
# Place the cannon
self.screenshot('place_the_cannon')
self.tap_image("queryimages/place_the_cannon.png", width_modifier=0.75)
sleep(2)
self.screenshot('finish_now')
# Use gem to finish right away
self.tap_image("queryimages/finish_now.png")
sleep(3)
# Bring it on!
self.screenshot('bring_it_on')
self.tap_image("queryimages/bring_it_on.png", height_modifier=0.75)
sleep(10)
self.screenshot('battle')
sleep(10)
self.screenshot('end_of_battle')
# To be continued…
if __name__ == '__main__':
unittest.main()
Now, let’s check out some of the stages in this test script. In this test_tutorial, you will find below given steps:
It can be easily figures out whether the test implements on iOS or Android (self.isAndroid()). Moreover, it seems UI content differently. In Android, it can be tried to search a button by using an element’s ID (element_name) and iOS by using an accessibility ID along with a description (“Okay”). Such thing can be checked for iOS’ Game Center login.
You can take screenshots at various steps and easily stored in files that entered as the parameter in a function call.
A check is completed after looking whether “goldmine” seems on screen by comparing two PNG files by using a self.find_image call. Once pictures are matched then script continues implementing the tutorial of the game.
Now, the tutorial continues with below given steps:
- Go to shop.
- Purchase cannon.
- Put the cannon.
You will find the information of the three items in those PNG files:
shopbutton.png, cannon.png and place_the_cannon.png.
At last, the tutorial completes and the battle starts. The application is closed after the battle.
With below given video, one can easily execute test at each step. Here, we are using three devices like iOS (iPhone 4S) and Android phones i.e. Samsung Galaxy S3 Mini and HTC One X. Click hereto see video on YouTube.
How to Use Image Recognition?
Image recognition is showed in this examplefor recognizing features like pixels and graphic content that are shown on screen and to compare the two different images. The algorithm is developed to identify pictures that were used on real devices and on two leading platforms like iOS and Android and it also used a single test script for both platforms.
Such type of image comparison is very convenient to identify UI elements and graphics, which are rotated and resized. Let’s say template image comes with some distinctive features like text can be easily abstracted from the background content. Moreover, feature-based recognition can also be used in some cases.
In this example, if the “Button” text has been resized or rotated, it can be easily identify to take further steps. The approach of comparing images is easily explained by below given functions.
def find_image(self, queryimage_file, screenshot_match=None):
log('Trying to find %s in current screen' % queryimage_file)
fd,screenshot_file = tempfile.mkstemp()
try:
os.close(fd)
# print screenshot_file
head, tail = os.path.split(screenshot_file)
self.screenshot(tail, path=head)
head,tail = os.path.split(queryimage_file)
screenshot_match='%s/%s-match.png' %
(self.screenshot_dir, tail.split('.')[0])
return ImageFinder.findImages(queryimage_file,
"%s.png" % screenshot_file,
scale_tolerance=0.9,
screenshot_match=screenshot_match)
finally:
os.remove(screenshot_file)
def screenshot(self, name, path=None):
full_path=None
if path:
full_path='%s/%s.png' % (path,name)
else:
full_path='%s/%s.png' % (self.screenshot_dir, name)
try:
self.get_driver().save_screenshot(full_path)
except WebDriverException: # for iOS, sometimes times out, so retry!
#log("Failed taking screenshot %s - retrying" % full_path)
self.get_driver().save_screenshot(full_path)
#
width,height = get_image_size(full_path)
if(height > width):
if self.platform_name == 'Android':
#log("Rotating screenshot 270 degrees")
os.system('sips -r 270 %s >/dev/null 2&>1' % full_path)
else:
#log("Rotating screenshot 90 degrees")
os.system('sips -r 90 %s >/dev/null 2&>1' % full_path)
def tap_image(self, query_image,
width_modifier=0.5, height_modifier=0.5, retries=2):
retries_left = retries
rect = None
while retries_left > 0 and not rect:
rect = self.find_image(query_image)
retries_left = retries_left - 1
image_name = os.path.split(query_image)[1]
self.assertIsNotNone(rect, "Image %s is on screen" % image_name)
x = rect[0]+rect[2]*width_modifier
y = rect[1]+rect[3]*height_modifier
log('%s button found at %s %s (%sx%s), tapping at %s %s' %
(image_name, rect[0], rect[1], rect[2], rect[3], x, y))
self.tap(x,y)
def tap(self, x, y):
log('Tapping at %s,%s' % (x,y))
driver = self.get_driver()
if self.isAndroid():
if self.isSelendroid():
touch_actions = TouchActions(driver)
touch_actions.tap_and_hold(x, y).release(x,y).perform()
else:
action = TouchAction(driver)
action.press(x=x, y=y).release().perform()
else: # iOS
action = TouchAction(driver)
# TODO: Temporary hack
# On iPhone 4S = res. 480 x 320 for taps but screenshots are 960 x 640
action.press(x=int(x/2), y=int(y/2)).release().perform()
Wrapping Text:
Testing is an important task to get success in the highly competitive landscape of mobile applications and games. If testing done poorly then it will take up to 20-50% of total development effort. In some case, it would take single budget cost of your budget.
Before starting with testing part, you should consider below given points like:
- Automate generic processes as much as possible by planning in proper manner.
- Choose proper testing technology according to app needs.
- Test automation is available 24/7.
- For global reach, it is advisable to use a cloud-based platform