Android App Usability Testing

Android App User Experience Testing

The mobile market is large and growing rapidly. It is estimated that there will be 4.5 billion mobile phone subscribers around the globe, which will exceed the global population.

Before you begin… A few words about this guide

This article is a step by step guide. This structure was created to give you completeness. Many articles will detail the process of testing android applications usability testing. These articles are fantastic (so much so that I have cited many of them in this guide), but they can be confusing for someone new to android user experience testing. This article is your solution.

A caution. Before you begin usability-testing on mobiles you need to know how to design mobile interfaces that are based on the best android usability practices. You should take the Mobile UX (UX) Design course if you don’t have it. It also comes with a industry-recognized Certificate. You can also buy a book about the subject.

Alright. Let’s get started. Let’s begin with a question.

Question no. 1 – Do I need to read every step? You can skip any section if you are confident in it. This index contains the major sections of this guide.

Question no. 2 – Is this a guide that can be used to test the usability of mobile sites? This testing method is intended for android mobile apps testing. This method can be used to test both mobile websites and mobile applications. Android application usability test was my focus, as android applications are very popular. Also, it seems that most articles or at least the ones I found tend to be more focused on mobile website testing.

Android Application Usability

Let me start this guide by asking you a question. What are the most common activities of phone users? Interacting with Android Mobile Applications (or Apps as we refer to them). You did read that right. Recent research shows that US phone users spend 86% on mobile apps. This figure was higher than 89% according to another study. Further, this study also found that android mobile users use only five apps out of 24 they usually use each month.

It is therefore not surprising that the number of smartphone applications has increased dramatically, especially for games and online media. Forbes estimates that nearly 270 billion apps will be downloaded by next year.

Despite this, android phone apps are still limited by the small screen sizes and limited performance capabilities on the devices they run. Smartphones have made great strides in terms of screen size and processing power. It’s also true that android apps design has improved significantly. They lack the processing power and screen size of larger devices like laptops or desktops.

Research has shown that usability plays a key role in the success of android applications. In reality, is a popular trend among mobile phone apps that users perceive as easy-to-learn, user-friendly, and more efficient in completing tasks. Another researcher has identified a direct connection between android app usability and user acceptance.

Although android application usability is vital, there are not many guidelines. Usability Testing is the best way to assess the usability of android apps.

Android Application Usability Testing

Before you begin thinking about testing and how you will do it, you must create a usability plan. This is the blueprint for the test. Although there is no set structure, the majority of it contains the following sections (Rubin and al. 2008):

  1. The test purpose, goals, and objectives
  2. Research questions
  3. Participant characteristics
  4. Method (test design)
  5. Task list
  6. Test Environment, Equipment and Logistics
  7. Test facilitator role
  8. Data collection and evaluation
  9. Presentation and content of reports

The usability testing plan shows that the usability of android applications requires the following:

  1. The test objectives
  2. Tasks to be completed
  3. Test Documents (content form, orientation script, pre & post-test questionnaires)
  4. Test Participants
  5. Test Method

1. The Usability Testing Objectives

Setting goals is the first step in any usability test session. What are your goals for the usability test session? With the HTML3_ test HTML3_, what hypothesis are you trying to test?

How can one set goals? This can be done in many ways. One of my favorite techniques is the variation, as described by Michael Margolis at Google Ventures. This involves asking questions to app’s stakeholders (and the developers behind it) in order to identify important areas.

  • The app’s roadmap
  • The app targets users and markets
  • Competitors of the app
  • It is possible to do research that has been done before and any additional research that the team needs.
  • The potential impact of the research described above
  • Timing and scope

These interviews will provide two important insights:

  1. What stakeholders already know
  2. What they’d like to know

It is easier to identify the goals and the usability metrics to measure them based on the answers to the questions.

It is important to identify the goals:

  • Specific
  • Measurable
  • Prioritized

2. These are the Tasks to be Completed

After the objectives are set, it’s time to move on to the next step, setting tasks. Tasks should be one sentence in length and contain the interactions that the test users need to perform, such as:

  • Register for an account
  • Sign into your account
  • Upload a photo
  • Accept friend requests

Instead of asking the test users to complete a task, and making them feel like they are being tested for something, task should be transformed to task scenarios. These scenarios provide context and make it easier for participants to understand why they are doing the task. They also allow users to interact with your application in a more natural way. The task scenarios should be:

  • The steps are concrete, practical and easy to follow.
  • The order of the test session is important to ensure smooth operation
  • Tied to one or several objectives.

It is important to conduct a dry run of usability test in order to make sure that the tasks are completed as intended.

3. The Usability Test Docs

When testing usability testing, you will need a variety of documents. Although the content and number of documents may vary, the most common ones are the following:

  • Consent Form: (for Minors and Adults).
  • Post-test questionnaire. This questionnaire’s content can be anything from the two post-test questions suggested by Uxpert Jared Spool, to more standard questionnaires like the Post-Study Usability Questionnaire or the System Usability Scale.

For more usability testing documents you may find helpful, please go to the website .

4. The Test Participants

The next section will discuss the android application usability testing method. This is a user-oriented technique that involves real users performing realistic tasks that the app is supposed to accomplish. While testing with real users takes more resources, it yields more accurate results.

Raluca Buidu, from Nielsen Norman Group, recommends that test participants be recruited who have used their devices at least 3 months. This will eliminate any problems that may arise from the use of the device, rather than the app. Additionally, test participants’ devices can reveal more issues because they are using real devices instead of top-of the-range devices and fast internet connections which are often available in development environments.

When selecting participants for the usability test , there are many things to consider. Participants must:

  • Represent the intended users of the app (i.e. The target users
  • Use the Operating System of the device on which the app is to be run (including any version/s).
  • Consider the current stage of the application. It could be an app in its early stages or one that is already on the market. This could make it crucial to consider factors like confidentiality and the experience that the test participants have in dealing with mobile apps.
  • You must be available for the usability tests at the appropriate time, place, and frequency
  • Accept the terms of compensation you offer (if applicable).
  • Sign a usability consent form

To ensure the recruitment of the best test participants, it is a good idea to create user personas. These can then be used to screen potential test participants to identify the most qualified candidates. If you have potential test participants (e.g. You can either use your registered users or perform the recruitment yourself.

There are many agencies that can help you outsource this task. You can choose which one you prefer, as each method has its pros and cons. The Nielsen Norman Group conducted an interesting survey and found the following interesting results.

  • Many companies employ their test participants
  • The cost of recruiting agencies is quite high (approximately $84 per individual if you are recruiting students or average consumers, and $161 per participant if you are recruiting professionals with high-end skills).
  • Employers who recruit their own participants average 1.15 hours of staff time per participant

These articles provide more information on how to recruit participants to usability studies.

  • How to recruit participants for Usability Studies By Jakob Nielsen and Hinderer Sova
  • Recruiting Testing Participants by
  • 6 Methods to Recruit Participants for Remote Usability Testing By Natalia Bruzi
  • 7 Methods to Find Users for Usability Testing By Jeff Sauro

5. The Android Application Usability Test Methodology

Two main methods are available to conduct usability tests of android applications. Each method has its advantages and disadvantages, it is clear. These are:

  1. Laboratory-based usability testing
  2. Remote testing usability

This article will use laboratory-based usability testing. This method allows mobile applications to be tested by real users on real devices. It allows the evaluator, whose role will be explained further below, to have full control over the test and can set the tasks. This allows him/her test all usability aspects. The results are also more precise and easy to understand. It is recommended by many usability experts, such as Raluca Budu from Nielsen Norman Group or Jeff Sauro of MeasuringU as the best usability testing method to mobile.

The ‘Components of a Lab-based testing

Lab-based usability tests involve the observation of participant participants performing tasks with a mobile device, in a similar setup to the one below: The typical setup for usability test on android.

You can break down the above arrangement into these ‘components:

A) Camera

There are two types of camera setups. Both will capture users’ gestures.

  1. Document/fixed-position cameras: A camera is located at a fixed point. (e.g. IPEVO’s Ziggi HD Plus
  2. Mobile testing sleds / Cradle-based cameras: The camera attaches to the cradle where the device is placed. When performing the tasks, the user holds the cradle in their hand. There are many cameras to choose from, such as Mr. Tappy, MOD 1000 and many others. You can also find many DIY setups like this one by Rik Williams, who assembled a test camera sled in 15 minutes using materials readily available in any office. Harry Brignull built a testing sled at a cost of less than $8 . While DIY setups can be relatively inexpensive and quick to make, there are limitations that could affect the testing process such as extra weight, less durability and portability and a slower build time than professional testing sleds. A cradle-based camera allows users to use their smartphone in natural motion, which is a major advantage over fixed-position cameras that force the device to stay on a flat surface. However, the extra weight and size of cradle-based testing devices is often criticized. However, the increased use of lightweight materials has reduced this.

Wireless testing Colman Walsh, a Smashing Magazine contributor, describes a cost-effective and interesting setup that makes use Apple’s AirPlay technology. It is completely invisible to the test participant, unlike document- and cradle based systems. The article, although it is not an official testing method, is well worth reading.

B) Webcam (Optional)

An alternative view can be provided by using a webcam that is pointed at the user’s face. This will allow the interviewer to capture facial expressions and help them understand the interaction.

C) Recording software

Recording software is required to project the session onto the facilitator’s screen, and possibly record it as well. Software that allows the session to be displayed in real time on multiple screens over a network can also be used. This is especially useful when there are multiple stakeholders. Document cameras and cradle cameras often come with their own software. There are many great android application testing tools available that can be used if that is not the case. Jeff Sauro, MeasuringU uses GoTo Meeting for audio and video to be saved as a compressed and shareable.wmv files and Camtasia to save webcam video. You can also use Morae or WebEx , but the cost for this license is quite high.

D) Test venue

It was traditionally done in specialized usability test labs. However, it is now possible to conduct this type of testing with sophisticated technology at a low price. It is important that you have adequate lighting, and not too much above the device being tested. This can create a glare.

E) Facilitator

The facilitator’s role is to facilitate testing. This means that they address any issues the test participants may have regarding the task or the device they are using. The facilitator should be a usability specialist. In addition to that, they must monitor the test on their screen and ensure that the test participants complete the tasks.

The Testing Procedure

As shown below, the testing process can be broken down into six steps. Each step’s time is calculated based on the average time each step takes according to UserFocus. Each session will take around an hour if one follows the suggested times.

1. Signing the consent form / Welcome5 minutes
2. Interview for Pre-HT0_ Test5 minutes
3. The task test35 minutes
4. Post-test questionnaire5 minutes
5. Interview after test5 minutes
6. Debriefing5 minutes

All the steps above have been explained earlier. Step 6 – debriefing – is a little less well known. The Test session ends with debriefing . This involves reviewing and analysing the actions of the participant. The participant is able to share their experiences during the debriefing session, which provides further insight into why they did what they did. The session may indicate the problem, but the debriefing session provides insight into why the problems occurred.

Reporting the Android Usability Testing Results

After the usability testing sessions are completed, it is time to review all data, compile and analyze it so that it contains useful recommendations.

First, it is important to split the data between quantitative and qualitative data.

  • Quantitative data from the testing session can be used to calculate usability metrics like completion rates, success rate, task times and satisfaction ratings.
  • Qualitative data can be compiled to give insights into the participants’ paths, problems encountered, and the answers they gave in the questionnaire, post-test interview, and debriefing sessions.

While there isn’t a standard structure for reporting, generally recommends the following sections:

  1. Background Summary: Information about the application being tested, its location, equipment used and who was the testing group.
  2. Methodology The test was conducted, the task scenarios and the metrics that were collected. Include information about the test participants (brief summaries demographic data – don’t release participant names).
  3. Test Results: This report contains both qualitative and quantitative data, as well as an analysis.
  4. Conclusions and recommendations: Based upon the observations made at point 3, (i.e. Provide a list of recommendations to be implemented to address any usability issues identified. But don’t just report the issues. Highlight any findings that demonstrated good usability.

Wrap it up

You can see that there is a structure to android app testing. Even better, the approach can be modified to suit the needs of each application being tested as well as any stakeholder interested in the results. It is well-known that usability testing is crucial, particularly for mobile, due to the wide variety of devices and specs. There is no reason why usability testing should be done in this day and age.