Integrating Usability Testing into Agile Software Development

This is article regarding real time usability testing in Agile Development:


The best way to validate design and requirement decisions with your users is via usability testing. The general idea is to gather feedback on early concepts, understand workflows & user pain points and allow designers and product managers to document findings and create/maintain personas, roles and system workflows. 

Test Subjects

As a general rule, the best test subjects are real product users, but it is better to test anyonerather than not to test at all. We’ve created 3 tiers of participants:

  • Tier 1: Customers/Partners – Ideally, they only test real users from our customers and partners in the field. We are establishing a charter customer program to make real users readily available for such testing.
  • Tier 2: Recruited Participants – When needed, the team may supplement their testing with recruited participants – either paid, gifted or volunteer.
  • Tier 3: Internal Employees – As a last resort if no other participants are available, we’ll test our ideas on Sales, Marketing, Business Development, and/or any other customer-facing groups. This approach still provides valuable feedback and validation, but we need to keep in mind that results may be slightly skewed.

Planning, Preparation and Facilitation

To prepare for the test, the design team collaborates closely with product management to define the usability tasks to test and the interview questions concerning value. We try to concentrate on the key tasks that users will do most of the time.

How they test users is determined by multiple factors, including what type of test environments are needed (i.e., Web v. Mobile), as well as when and where we are testing. Over the years I’ve tested in-house, remotely (using a video conferencing tool like Adobe ConnectJoinMe orSkype) and on-site at a customer’s office. The latter is usually the best case since you can get a feel for the user’s actual workspace limitations, workflows and peer interactions – but it is not always possible, so take what you can get.

We also consult with the product team to decide the most efficient approach, based on the available time and resources needed to:

  • Create a wireframe or build an interactive prototype
  • Present working code in a test environment
  • Use card sorting, paper prototypes and/or taxonomy studies.

When conducting the test, it’s important to make sure to provide the user with a proper orientation to the environment and ask for permission to record. It also helps to mention that that they won’t be hurting your feelings by giving an honest opinion, and to continually remind them to “think aloud” to provide feedback and context. Watch what they do, and observe both verbal and non-verbal clues about why they fail or become confused.

The testing environment and equipment (laptop, phone, tablet, software) may vary, depending on the product type (desktop v. mobile), but should be a quiet, closed space that can minimally accommodate a tester, a facilitator and a note taker. Recording audio and video is highly recommended; using cameras to capture both screen and the user’s face provides emotion and context to any issues that arise. A laptop equipped with a webcam and screen recording software is ideal — I typically use Silverback on a MacBook Pro for this. Remember that face-to-face testing is ideal, but any testing is better than no testing.

When to Test

The  recommendation is to test users early and often, during all phases of the product lifecycle. This includes:

Conceptual/Prototyping Stages – This happens earlier in the project, during the requirements/planning stage (iteration -1, no code or design specifications). Try to recruit 8-10 testers to participate in 45-60 minute individual sessions covering multiple features or a new application. It’s critical to get input from existing customers for validation and understanding real workflows via wireframes and/or an interactive prototype. This type of testing usually requires a detailed script and covers a new product or multiple user stories. In the past, I’ve successfully used FlashHTML, Axure and PowerPoint to create mid-to-high fidelity interactive prototypes. I’ve found that the choice of tool is less important than its ability to simulate an experience that mimics the desired end product, or that it can be delivered within the chosen test platform – and you absolutely must be able to work efficiently with the tool.

Development Stages – During product development, testing is done during each sprint. Note that the duration of the sprint isn’t important, as long as at least one round of testing is conducted per sprint. Usability testing is much lighter and informal during this stage, so try to recruit about 5 participants for 15-30 minute individual sessions that focus on a specific feature. You’ll be able to quickly spot trends and patterns in usage, which will allow you to iterate on your design during the sprint, if needed. Tests can either use live code (from a QA environment) or a quick wireframe/mockup if testing items for a future sprint. Allocate a set amount of hours for usability testing activity in the sprint backlog – to be burned down – making sure that all UT activities (planning, testing and analysis) fit within your time estimate.

Usability Testing Process Graphic

After testing is completed, we generate a Usability Observation List that will be shared with the team (via verbal review at scrum, and entered into our wiki within JIRA). Product Management will prioritize these results into one of two categories: Bugs or Enhancement Requests. Bugs are entered into the tracking system (we use JIRA, but TFS, or even something as simple as Excelworks as well) and should be fixed before the end of the sprint. See figure 1 for a visual representation of our process.

Post Launch – After your product is launched, be sure to have an internal retrospect with the product/feature team to discuss what was successful versus what was a problem and consider how to streamline and improve your UT process. In the past, I’ve used surveys, email discussions, message boards and field-testing to gather feedback from end-users. It’s also helpful to have the product team speak directly with external product champions about adoption rates and to ensure reference-ability for marketing and future UT sessions. Lastly, but most importantly, reach out to your charter customers to follow up on the end results now that they are using features in their daily workflows.

Final Deliverables

Our deliverables vary depending on what type of testing has been done. For conceptual testing, deliverables typically include a Summary & Recommendations document that may include a deep dive to identify what the core root causes were for failures based on actual observations, conversations and concrete recommendations to improve the user experience. Recommendations are categorized into UrgentImportant or Nice-to-Have to help the product team accurately prioritize, and the overall scores, statistics and notes are presented.  This document is uploaded into the wiki, along with any audio/video recordings to be archived for reference. If you have the resources to convert the recordings into a transcript, it can be quite helpful for easily searching and quickly scanning.

Deliverables during sprint testing usually include the Usability Observation List that quickly identifies the points of failure and provides recommendations to improve the user experience. These findings are communicated in the daily scrum and uploaded to the wiki for future reference (along with any audio/video recordings).

Finally, the post-launch deliverables can include a retrospect meeting, updating the feature enhancement list based on customer feedback, and identifying reference-able customers. This last deliverable can lead to highlighting a customer in a case study, or inviting interested customers to participate in future UT sessions.


As you can see, user testing is an invaluable part of the agile methodology that can help you better understand your customer’s needs and make your products and apps more useful to users. No matter how or when you test, you’ll reap many benefits such as early problem detection, increased user satisfaction, reduced support costs and increased efficiency for users. And you’ll constantly be amazed by the innovative ideas for enhancements and new features that come directly from your customers.

The bottom line is that if you are developing in agile, then you should be incorporating some form of usability testing into your iterative process. It does not need to be a formal or expensive process – it just needs to capture user feedback in some way so that you can ensure a usable product. As Admiral Grace Hopper once said, “One accurate measurement is worth more than a thousand expert opinions.”



Agile Usability Engineering: by Thomas Memmel.

Agile Usability Engineering is a concept to describe a combination of methods and practices of agile development and usability engineering. Therefore, this entry commences with a brief note on agile methods.

In recent years, agile methods for software and web engineering have reached widespread acceptance in the community. In contrary to classic, heavy-weight software engineering processes like the V-model, agile methods (Ambler 2002) begin coding very early while having a shorter requirements engineering up-front as well as less documentation. Following the paradigm of Extreme Programming (Beck 1999), implementation of code takes place in small increments and iterations, and small releases are delivered to the customer after each development cycle. During a short claims analysis, called the exploration phase, the development team writes user stories trying to describe user needs and roles; the interviewed people need not necessarily be the real users of the later software product. Seen from a human-computer engineering perspective, Extreme Programming (XP) thus often fails to collect real user data and starts coding with just assumptions about user needs. The development in small increments may work properly as long as the software has no focus on the user interface (UI). Changes to software architecture most often have no impact on what the user sees and interacts with.

With the UI, it’s a different story. When designing UIs like websites, continuous changes of the user interface due to fast iterative design may conflict with user expectations and learnability, provoke inconsistency and possibly lead to user dissatisfaction. Evaluation of small releases with stakeholder participation does not ensure that the whole system provides a consistent conceptual, navigational or content model.

Nevertheless, the numerous discussions about agile approaches to user interface design (UID) have lead to a movement in the human-computer interaction community, which has begun to reconsider its user-centered heavy-weight usability lifecycles (see table 1, compare Mayhew 1999).

Heavy-Weight Processes

Light-Weight Processes

Detailed, up-to-date documentations and models

Cards and hand-drawn abstract models. Travel light. Communicate rather than document.

High-fidelity prototypes

Abstract prototypes, use simplest tools

Develop and prove concepts with user feedback. Iterate.

Courage. Design for needs (user’s tasks) rather than user expectations. Retrieve design from models rather than continuous user feedback.

Time-consuming usability evaluations, workshops with intense stakeholder integration

Fast usability inspections. No need to evaluate if models are right.

Table 1 : Comparison of heavy- and light-weight processes exemplified by a few chosen aspects of principles and methods.

Increasingly, software engineering (SE) and UID have to cope with a shorter time-to-market, whereas the quality of the delivered software must not suffer. This continuous shortening of development lifecycles is a great challenge to both project management and the applied methods and tools. Therefore, many usability engineering (UE), UID and agile method experts have developed light-weight approaches or so-called agile usability engineering (AUE).

An example of an agile approach to user interface design

Constantine and Lockwood (1999) were one of the first and most important practitioners to come up with an alternative, light-weight approach to UID (see table 1). They believe that relying too heavily on user needs and feedback may lead to an overhasty narrowing of the design space. Designers run the risk of designing what the users wish as opposed to what they really need. In their opinion, iterative rapid prototyping is like trial-and-error design and may never lead to an optimal solution if too many stakeholders are involved and too many opinions influence the design process. Instead, their usage-centered design approach is based upon abstract models to describe users, tasks and content. Users are still involved as resources of information and validation, but the design is a straightforward transformation of models into design. Usage-centered design may therefore be described as a design by engineering approach. Due to the usage of easy-to-understand and easy-to-manage artifacts like hand-drawings or simple cards for documentation and abstract low-fidelity prototypes, usage-centered design can be rated as one of the first agile approaches to UID.

User-Centered Design

Usage-Centered Design

Focus is on users: user experience and user satisfaction

Driveb by user input

Substantial user involvement

  • User studies
  • Participatory design
  • User feedback
  • User testing

Design by iterative prototyping

Highly varied, informal, or unspecified processes

Design by trial-and-error, evolution

Focus is on usage: improved toos supporting task accomplishment

Driven by models and modeling

Selective user involvement

  • Explorative modeling
  • Model variation
  • Usability inspections

Design by modeling

Systematic, fully specified process

Design by engineering 

Table 2 : Comparison of user- and usage-centered design. Source: Constantine and Lockwood (2002)

Whereas Constantine and Lockwood defined their design approach based on modeling, others, like Gundelsweiler et al. (2004), have developed approaches based on similarities between XP and agile methods on one hand, and user experience (UE) on the other. Gundelsweiler et al. (2004) presents an agile user-centered approach to software engineering and proves that the incremental and iterative procedures of XP, and other related processes of agile methods, can be brought together with current UE practices. Gundelsweiler et al. (2004) integrates both methods of user-centered design (see e.g. Norman & Draper 1986) and usage-centered design (e.g. Constantine & Lockwood 1999). Other examples of combining XP and UE are Obendorf et al. (2006) and Holzinger & Slany (2006). They term their approaches XPnUE and eXtreme Usability (XU).


Cross-Cultural Considerations for User Interface Design

A comprehensive overview on Mcdonald’s websites based on culture diversity:-

Standardization vs. Localization:

A couple of weeks ago, while doing some cross-cultural research on websites, I stumbled onto several McDonald’s’ websites in different countries. It was particularly striking that each of the sites was radically different, not just in terms of text and translation, but from the kind of images used, typography selected, color palettes designed, layouts chosen, to even the favicons!

As looking for some more examples from different countries, it appeared that the European websites were more similar visually when compared to the set of Asian websites. While I debated whether it was a good design strategy or stereotyping of cultures, it was clear that McDonald’s felt it was important.

On the one hand, the user interface design process focuses wholly on the user and his context — his needs, wants, expectations and preferences — and on the other hand, global businesses want to cater to and engage with consumers across the world and consider demographic diversity which, while possible, can get very complicated.

McDonald’s – India (upper left)
McDonald’s – Pakistan (upper right)
McDonald’s – Germany (lower left)
McDonald’s – United States (lower right)

Imagine your favorite e-commerce website being visited by users from different parts of the world, assuming that language translation is successful; are there elements of it that might make it difficult for them to navigate through the site? Are there metaphors that they simply would not understand? How about mental models; do users from different cultures look for different data to make decisions? There are several other questions that illustrate the depth of this problem. But the bottom line is to help make communication effective between a global business and a local user. It simply isn’t enough to use a standardized website that translates the text and utilizes the same images, layout and color of the parent site (Becker and Crespo, 2001). For instance, consider the OK gesture; in most English-speaking countries, it is a sign of everything working well, but in several European countries the same gesture means “zero” or “worthless.”

The debate around standardization versus localization in website design continues to be a bone of contention. While some argue that due to the nature of the internet, cultural distance will minimize and we will see a homogeneous online culture. Many other research studies show that consumers prefer to visit and interact with sites that are made specifically for them and which contribute to online customer trust, satisfaction and e-loyalty (Cyr, 2004). Even so, the goal of culturally sensitive usability is to understand different cultural perspectives holistically with the user-centered design approach, and to avoid stereotyping cultures or making tacit assumptions about politically correct design.

Research on cultural differences:

Much research around cultural differences has been published. Few of these works are well known, with particular focus on Fons Trompenaars and Geert Hofstede’s work.

Trompenaars, a prolific Dutch researcher in the field of cross-cultural communication, with Hampden-Turner, studied how people from different cultures solve problems. Hofstede, an influential Dutch cultural anthropologist and psychologist, conducted extensive and exhaustive interviews with IBM employees in 53 countries in the early 1970s, with over 100,000 data points. Both Trompenaars and Hofstede discovered that there were significant differences among cultures in different countries. Both went on to define culture as a set of shared characteristics such as thoughts, values and behaviors. Based on their research, Trompenaars and Hofstede formulated different theories on cultural dimensions, which defined sharp lines of distinctions along which cultures differ.

Given the global platform of the Internet where social experiences are shared and communal interaction occurs, it becomes critical for web and web-based application design to consider culturally diverse user groups. While some fundamental principles of usability, such as simplicity and consistency, are elemental, in such situations there are certain design characteristics that need to be reflected upon.

According to Cyr (2004), there are critical characteristics of design that need attention in a cultural context:

  • Language, which is a distinctive aspect of different cultures, which when moved to the online domain presents its own challenges of quality of translation, representation and stylistic elements.
  • Layout that serves as a communication bridge between the user and the system, it involves placement of banners, menu items, orientation, amongst others.
  • Symbols refer to “metaphors” denoting actions for the user and vary from culture to culture. They may be icons used for currencies, locations, and other navigational elements.
  • Content or structure, referring information or features that the site offers and its organization. 
    Navigation that is easy to use and facilitates access to information.
  • Multimedia, including video, animation, images and sound.
  • Color – where color semiotics varies across cultures and can impact user expectations

Hofstede’s Five Dimensions:

The concept of culturally sensitive design includes all these characteristics as well as the emotions, behaviors and ways of thinking of the individuals also considered as users (Eristi, 2009). As per Eristi, Hofstede’s cultural dimension theory seems to be effective in the culturalization process of appeal in web design.

According to Hofstede (1980), world cultures vary across certain consistent and fundamental values. These 5 dimensions are:

  • Power distance
  • Collectivism vs. individualism
  • Femininity vs. masculinity
  • Uncertainty avoidance
  • Long vs. short-term orientation

Marcus and Gould (2000) make several inferences from Hofstede’s theory and how it can be adapted to cultural differences. Scientists have continued to build upon their work, creating many ‘best’ practices for cross-cultural interface design.

Let’s take a very brief look at each of Hofstede’s dimensions and preliminary implications for user-interface and web design to make design characteristics culturally relevant:

Power Distance (PD):

This is the extent to which less powerful members of organizations accept that power is distributed unequally. High power distance countries are more autocratic; low power distance countries are more democratic.

Marcus and Gould mention that high power distance countries have a stronger focus on expertise, authority, experts, certifications, official stamps and logos, amongst others. Access to information is restricted and social roles are used to organize information (managers’ section vs. non-managers’ section).

  • Introducing visuals, expressions, images with authoritative body language, that are informative and guiding will work well in high power distance countries. Low power distance cultures may prefer activities of daily life, popular images, symbols and colors.
  • Clear indicators of navigation that present all information at once are appreciated in high power distance countries, whereas less dense and informally organized structure is preferred in low power distance countries.
  • High power distance cultures tend to prefer sites with a formal layout but with more vivid visuals. Limited choices and restricted access to information is favored. Visual elements that are resonant with the pulse of the nation are valued.
  • Low power distance countries prefer informal layouts with clear access, and multiple choices on use. Flexible interactions that guide in case of errors will be valued. Visual elements that are more universal are preferred.

Coca-Cola – KuwaitCoca-Cola – Ireland

Kuwait has an index of 90 on the dimension of power distance, meaning that it values hierarchy, top-down direction and centralized power. The information architecture is very organized with very little information for the press or the presence of a media center. There are very few choices people can make. It has a formal layout but can look “flashy” with the overbearing presence of red. The images used are powerful and used to guide the user into the site.

  1. Ireland has a lower power distance index of 28. They value informal structures where hierarchies are not maintained very strictly. The Coca-Cola Ireland site is very transparent in sharing information. The information architecture is shallow and clearly surfaced. Search functionality is provided for multiple choices on use and the site encourages exploration.

Individualism vs. Collectivism (IDV)

This refers to the degree to which individuals are integrated into groups. Individualistic societies emphasize personal achievements and individual rights. Collectivist societies emphasize group affiliations and loyalty.

In the same paper, Marcus and Gould (2000) believe that motivation is based on personal achievement in individualistic cultures, while collectivist societies value harmony and consensus to move forward.

  • Content based on materialism and consumerism will work well in individualistic cultures, and those that represent community and harmony will work better in collectivist societies.
  • Language, sound, videos and metaphors that emphasize individual successes will be preferred in high IDV countries; with larger successes and goals will matter more in low in IDV cultures.
  • Individualistic countries will value importance given to the youth, whereas social actors of experience and wise leaders will be preferred in collectivist cultures.
  • Sites that offer extrinsic rewards tend to be valued by cultures that are highly individualistic. – – Venezuela

With a score of 76 on the IDV index, Italy is a very individualistic culture. Personal achievement and a sense of self are important and this is highlighted in the Italian version of Only one image is used and portrays a young couple emphasizing the importance of youth.

Venezuela has a score of 12, and is among the most collectivist societies in the world. There is a high preference for belonging to a larger group and people are expected to be virtuous and act for the greater good. In the Venezuelan site for, this is reflected clearly with the number of people both young and middle-aged shown in the background. They represent community and harmony which is important to collectivist societies.

Femininity vs. Masculinity (MAS):

This talks about the distribution of emotional roles between genders. Masculine cultures are competitive, assertive, materialistic; feminine cultures place more value on relationships and quality of life.

High masculinity cultures place importance on traditional gender, family and age distinctions, whereas feminine cultures emphasize the blurring of gender roles.

  • Visuals with a feminine theme including family will be preferred in highly feminine countries, and male dominated cultures will prefer images related to competition, meetings, success, personal gratification, etc.
  • Symbolic indicators and metaphors of achievement, results, and objectives will be preferred in high MAS countries, whereas social activity and interaction will be valued in low MAS cultures.

KFC – AustriaKFC – Denmark

With a MAS score of 79, Austria is a high masculinity culture. The society is decisive, and values success and personal gratification. This is seen in the KFC Austria site, where winners and dreams are clearly surfaced on the homepage.

On the other hand Denmark has a score of 16 on the scale, making it a feminine culture. Such societies value caring for others and quality of life. The KFC site for Denmark makes an apparent differentiation with the softer colors and feminine icons and metaphors. The homepage highlights the video where social activity and caring for society is prominent. The visuals used also suggest consuming the meal with friends.

 Uncertainty Avoidance (UA):

This refers to the degree of tolerance for uncertainty and ambiguity. High uncertainty avoidance cultures are more emotional, and control changes with rules, laws and regulations. Low uncertainty avoidance cultures are more pragmatic, and have as few rules as possible.

Cultures with high uncertainty avoidance would emphasize simplicity, clear metaphors, limited choices and restricted amounts of data, according to Marcus and Gould (2000).

  • Content and visuals that are clear and concretely associated with lives of individuals will be preferred greatly by high UA cultures. More general and symbolic messages with less detail will be valued by low UA cultures.
  • Redundant cues in color and typography that increase ambiguity are shunned by cultures with a high uncertainty avoidance index. They prefer very clean and distinct choices.

T-Mobile – GermanyT-Mobile – Singapore

With a score of 112, Germany is one of the countries that fall high on uncertainty avoidance. According to Hofstede, given their philosophical heritage, they have a strong preference for deductive rather than inductive reasoning. A systematic overview needs to be given in order to proceed. The T-Mobile site provides plenty of information; visuals clearly indicate the message, and typography and colors cues are not redundant and help reduce ambiguity.

Singapore has a score of 8 on the UA index meaning that people maintain decorum and follow rules but not because of the need for a structure but because they fall high on the power distance index. Messages on the T-mobile site are very clinical. The information architecture is very shallow, not dense with detail. Several links are repeated many times within the homepage

Long vs. Short-term Orientation (LTO):

This refers to the time horizon of a society. Long-term oriented societies are oriented to the future, and are pragmatic, rewarding persistence and saving. Short-term oriented societies are oriented to the present and the past, rewarding reciprocity in social relations and the fulfilling of social obligations.

High LTO countries would value relationships as a source of information and credibility and content that is focused on practical value, whereas low LTO countries would prefer content based on truth, desire for immediate results and achievement of goals.

  • Content and visual elements that are focused on rules as a source of credibility are valued more in low LTO countries whereas high LTO cultures are oriented by the search of virtuous behavior.
  • Content and navigation that emphasize long-term development and goals of the business will be valued by high LTO cultures. Providing future plans and guiding the users to understand their future relationship with the brand will be greatly appreciated by high LTO cultures. Other symbolic indicators that focus on immediate gratification will be favored by low LTO countries.

Wrigley – ChinaWrigley – United Kingdom

With a score of 118 on the short-term vs. long-term orientation, China is one of the most long-term orientation cultures in the world. They value virtuous behavior and long term planning and commitment. For example, the information architecture of the site has surfaced their CSR initiatives on the homepage. The visuals and their slugs talk about responsibility, reciprocity and efficiency to create a better world.

On the other hand, the United Kingdom is one of the most short-term orientation cultures in the world. For example, Wrigley’s UK website clearly focuses on small but immediate gratification with the tagline in the image. The news aggregator widget mentions everything that’s new and so do the other images at the bottom. The layout is very clean and organizes information to allow the easiest access to information.


It is interesting to note how each of these dimensions can have a considerable effect on global web design. From shallow vs. deep information architectures, access to information, motivation and trust, security and personal information, navigation, simplicity, graphical elements, to even color, among others.

This becomes especially important in contemporary settings when e-loyalty is critical in online transactions, and visual appeal is one of the ways to increase trust. Human beings instinctively make snap judgments about everything they see and try to rationalize it afterward, and so it becomes essential to get a positive first impression.

The idea of website globalization has two complementary processes. As per Periera and Singh (2005), globalization consists of website internationalization and website localization that work in tandem. While internationalization refers to the back-end processes of creating modular and accessible global website templates, localization refers to the front-end customization where websites are adapted to meet expectations of a culturally diverse user group.

In the e-commerce environment especially, the market is truly global even for a start-up. It is necessary to have a framework that allows cross-cultural design and usability as part of the development life cycle of any website. There will soon be a need for creating tools that allow for cultural versioning to suit global businesses and their agenda to communicate effectively to the local user. Involving cultural factors in the design process increases the aesthetic value, perceivable quality and behavioral intention in the users.



Usability Testing: When to use remote usability testing

Remote usability testing offers greater audience diversity and higher fidelity to real-world user motivations than in-person methods (but should not entirely replace them). It is recommended recommend using ‘Moderated’ testing for understanding an issue and “Unmoderated” to get large volumes of data. 


Introduction – to remote usability testing:

Traditional in-person usability testing involves an evaluator and a participant being in the same location at the same time and looking at the same screen. The participant in the usability testing session is then asked to perform certain tasks and vocalize their thoughts. Most usability professionals will tell you that usability testing is an invaluable tool in identifying and understanding usability issues. The method does, however, have certain weaknesses – such as:

  • Personal investment – Participants are often not personally invested in the tasks, no matter how hard you try and recruit relevant groups (i.e. they are not really considering buying this brand of car, they are just pretending).
  • Geographical limitations – The evaluators have to be in the same room as the participant, which means any truly international usability testing will be quite expensive.

Remote usability testing is a technique which seeks to address these issues. It does this by observing real-world users as they voluntarily use your site for their own purposes. Remote usability testing usually involves:

  • Participant and evaluator being in different locations
  • Observation and communication being mediated by technology

Benefits – of moderated remote usability testing

Although each method of remote usability testing has different characteristics and is best suited to answering different kinds of questions (as discussed in the next section), they do all share some benefits. Here are the benefits of almost all types of remote usability testing:

  • Audience diversity – You can easily run usability testing sessions with participants from different regions and/or countries.
  • Technical environment – People will be using their ‘real-life’ devices in their normal environment. This could, of course, have a large impact on participants’ behavior.
  • Personal investment – You can recruit participants as they arrive on your site, or are about to start a task you want to study. This means they will be personally invested in what they’re doing and have ‘real world’ questions that they want answered.

Note – Ethnio is a tool (currently free) which allows you – with the addition of 1 line of JavaScript to a page – to present potential participants with a DHTML pop-up screener layer (that cannot be blocked by a pop-up blocker). You can control the questions in the screener and users’ responses are immediately available for follow-up.

Types – of remote usability testing

There are 3 main types of remote usability testing. Each has their strengths and weaknesses and may be appropriate to different projects at different times. The 3 types of remote usability testing are:

  • Moderated – Similar to in-person usability testing, this involves the evaluator watching the participant complete tasks using a screen-sharing tool, and asking them to vocalise their thoughts (over the phone).
  • Unmoderated – user reported – Participants enter feedback and answer page-specific questions in a browser frame as they navigate the web site. The results are then collated (automatically or manually).
  • Unmoderated – automated statistical – Evaluators define the idealised, error-free performance of a task. Data is then automatically collected on users’ actions and compared to this idealised model (either manually or automatically).

Making a choice – selecting a remote usability testing method

A good general guide is that Moderated is usually the best choice of remote usability testing method if you want to develop a deep understanding of participant behaviours. Due to Unmoderated’s potential to allow you to run 100s, if not 1000s of remote usability testing sessions quite cheaply, it is often used to identify potential problem areas (for further investigation through Moderated methods), or illustrating the scale of an issue identified during Moderated sessions.

Please note: ‘Unmoderated – automated statistical’ remote usability testing is best for simple and easily-definable tasks.

Moderated – issues with remote usability testing

Moderated remote usability testing is a great method for developing a deep understanding of participants’ behaviours. There are, however, some particular issues concerning this method of remote usability testing which you will have to consider before planning any remote usability testing session:

  • Tools for screensharing, recording & communicating – It is important to find tools that give you the data you want, but it’s equally as important to find tools which are easy for the participants to install and use. Another factor to consider is the bandwidth which a tool requires, as this may limit your potential participants. 
    Note: Most remote usability testing practitioners relate good experiences of using ‘Reverse Morae with GoToMeeting’, which allows you to use Morae’s features to tag participant behavior).
  • Facial expressions & body language – Non-verbal cues are crucial in real-world usability testing, as they often portray a participant’s un-vocalised thoughts/attitudes. Unfortunately, these are normally not present in remote usability testing.
  • Personal connection – It is often harder to establish rapport and trust with a participant during a remote usability testing session. For this reason, it is often worthwhile to spend a little time at the beginning of the session introducing yourself and ‘breaking the ice’.
  • Supporting users – Helping a user to overcome a problem (for example: when they don’t know what to do next) can be difficult within a remote usability testing session. It was found detailed instructions are better than taking keyboard-control (because this breaks the unity of the participant’s experience).

Unmoderated – issues with remote usability testing

When using Unmoderated remote usability testing methods, it is important to try and capture how successful the site was in supporting participants’ goals. The issue under consideration is this: participants might be able to find the page which should answer their question, but does the page itself actually succeed in this goal?

One of the best ways of dealing with this is to ask participants a question, such as: “Which car has the largest trunk space?” It is an advise that you never rely solely on Satisfaction Surveys and always include such questions in your Unmoderated studies.

Please note: ‘Unmoderated – automated statistical’ remote usability testing is best for simple and easily-definable tasks.

Incentives – issues with remote usability testing

It has found that the most convenient incentive to offer participants is an Amazon gift certificate. These certificates have the advantage of covering a wide range of goods and only requiring the participant to provide their email address.

Summary – remote usability testing

Remote usability testing offers several advantages over in-person usability testing. These include a greater degree of audience diversity and a higher fidelity to real-world users’ technical environments and motivations. Moderated remote usability testing is best-suited to developing a deep understanding of an issue, whereas Unmoderated is best for getting large volumes of data.

The Moderated method of remote usability testing can not, however, be considered to completely replace in-person usability testing because of its limitations in the personal interactions between evaluator and participant. Researcher do, however, believe that remote usability testing represents a very cost-effective opportunity to explore a site’s usability.