Unboxing: remote usability testing

About this post

This is one of a special series of unboxing posts exploring the different options available for usability testing, covering the relative benefits of each as well as when to choose one approach over the other. From low-cost rapid remote testing through to moderated lab-based testing, there is no reason why some form of iterative improvement through user testing shouldn’t be a key part of every one of your digital projects, in order to create best-of-breed user experiences that deliver the highest levels of customer satisfaction and return on investment. 

What is remote usability testing?

Remote usability testing is a way of quickly evaluating platforms such as websites, mobile devices and desktop apps with real users, anywhere in the world.

The two types of remote testing are moderated and unmoderated:

Moderated remote testing focuses on gathering rich and detailed information, with a facilitator guiding participants through a series of tasks and questions. As with lab-based testing, there is an opportunity at the end for the facilitator to discuss why certain issues may have occurred with the participants, for improved qualitative and contextual insights about user behaviour. Unlike lab-based testing however, participants can be from anywhere in the world, giving added flexibility when recruiting representative end-users, who may be unable to attend a test lab in person or hard to schedule. The participant uses third-party software to share their device’s screen with the facilitator and observers, and the ‘think-aloud’ protocol to offer their thoughts and feedback during the session over a phone or conference call.

Unmoderated remote testing is conducted by the participant performing a series of predetermined tasks without a facilitator present. They use their own device in order to access an online third-party tool or website to complete the test, and closely follow a predefined test plan containing tasks and questions. Their screen is recorded by special software, and audio via their webcam or mic. Participants are asked to talk aloud at all times so User Experience (UX) consultants can evaluate their behaviour when reviewing the sessions at a later date. The advantage of not having a facilitator present is that much higher volumes of tests can be completed in parallel, and in very short time frames, by participants anywhere in the world. Similarly, the providers of third-party usability testing services often have a large panel of participants ready and waiting to take tests, which also reduces timescales. There is often the opportunity to ask some simple survey-style questions at the end of the test, but these won’t deliver the same level of qualitative insights as a moderated test with a facilitator present (although carefully-designed test plans can help mitigate this to some degree).

How many participants take part in remote usability testing?

The number of participants taking part in moderated remote testing can vary from 6 to 12 but it is very much dictated by project scope.

As unmoderated remote testing involves no facilitator a greater number of sessions can be completed. Quite often it is the number of days available for analysis by the UX team that determines the number of tests, because each video will need to be watched and reviewed by a UX consultant, who will also need to create a written report created detailing findings, recommendations and metrics.

What deliverables will I get from remote testing?

Remote usability testing deliverables are dependent on the nature of a given project and whether or not a moderated or unmoderated approach is taken, but they typically include:

Before the test:

• A test plan document outlining the proposed schedule, participant profiles, tasks, questions, session scripts, key research objectives and forms

After the test:

• A video combining screen and participant audio and video recordings
• A summary report and/or stakeholder presentation covering key findings, participant comments, recommendations and next steps
• A detailed findings report covering all findings from minor to severe, usability metrics such as time on task and completion rates, and participant comments, recommendations and next steps

When should I use remote usability testing?

Moderated remote testing is particularly useful if suitable participants are based overseas or are unable to attend lab-based testing in person, and especially if you need to use people with specialist knowledge or experience. The number of participants you can involve will be fewer than unmoderated testing, but the level of contextual insight about user behaviour and expectations during the test is typically higher.

Unmoderated remote testing is perfect for gathering larger volumes of user insight in very short timescales, so lends itself perfectly to iterative testing during sprints in an Agile project, for example. Similarly, larger volumes of testing data are often useful when making a business case for change, so unmoderated testing could also be used very early in the process to benchmark and prioritise improvements to an existing service. Many third-party providers of unmoderated usability testing now also offer the option for mobile testing (and apps) so you are no longer limited to just testing websites.

Example projects

Remote moderated

A website with a Middle-Eastern audience needed to be tested. The client recruited locally in Saudi Arabia so third-party software was used to screen-share and facilitate each session. Culturally Saudi Arabia and the UK are very different so recruiting locally meant the results were a lot more insightful and accurate than if the testing had been conducted in a UK-based lab environment.

Remote unmoderated

A client wanted an expert review of their website to be conducted but it was felt user testing would be more appropriate. A test link was included on their homepage and end-users asked to take part. A high number of people completed the test and their times, completion rates and comments were recorded.

Considerations

As mentioned above, moderated remote testing is great for getting insight from participants who might otherwise find it difficult, or impossible, to take part in usability testing. Since sessions are moderated by a facilitator, this also means that alongside the opportunity for improved qualitative insight there is more flexibility during the test; for example, you can even test with incomplete or early prototypes using cognitive walkthroughs. They do however rely heavily on a good phone and internet connection, otherwise slow screen-sharing or poor audio quality may impact testing significantly. Unlike lab-based moderated testing you may also miss some important cues from the participants’ facial expressions and body language.

Unmoderated remote testing can sometimes be a little less predictable, because the participants complete tests by themselves without the facilitator to help lead them when required, and the use of a predetermined test plan means less flexibility compared to a moderated test (where the facilitator can ask on-the-spot questions or suggest tasks not documented in the test plan). Common problems might include poor participant commentary, interruptions during a testing session, participants misinterpreting even the most carefully worded test plan questions or skipping some completely. In some cases this could lead to tests having to be retaken, so make sure you have tested, reviewed and refined your script before you roll it out to a mass test base. The overriding and significant benefit, though, is the ability to test very quickly, with much larger volumes of participants.

At Box UK we have a strong team of UX consultants with hundreds of hours’ testing experience. If you’re interested in finding out more about how we can help you, contact us on +44 (0)20 7439 1900 or email ux@boxuk.com.