How to Plan, Design, and Conduct Effective Tests – Jeffrey Rubin & Dana Chisnell

Date : November 12, 2009

“What Is Usability Testing?”

Rubin & Chisnell define ‘usability testing’ as “a process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.” (p.21) For them, the overall methodology is the same, but each test will have different results depending on the stage at which a product is tested and the type of goals and concerns a development team has for testing. They propose 4 types of informal tests that can be employed iteratively and that correspond to the different stages of the product at test and respond to issues of: exploration, assessment, validation, and comparison.

Why Test? Goals of Testing

Testing has two main objectives: on the marketing level, it aims at improving sales; on a user-centered level, it aims at minimizing user frustrations and maximizing a product’s usability. As the authors point out, testing goals inform the design of a product; those work in terms of the usefulness or relevance, learnability, efficiency, effectiveness, and satisfaction factors.

Basics of the Methodology

The authors emphasize the need for ‘tight controls’ when screening participants. This means making sure you recruit participants with similar backgrounds, such as: novice computer users. Assuming that you’ve picked the right participants for each test, testing will help you collect quantitative data that determine usability problems, causes and potential solutions.

Each test should involve: a purpose (goals and concerns), a participant (or two), an environment (i.e. a lab or an office room), observation method (a camera and observer who take notes), a methodology (interview or survey questions, and key tasks), data collection (qualitative and quantitative performance measures), results-debriefing (determining solutions for further implementations of the product).

Rubin & Chisnell make it a point that testing is not the ultimate answer, but is always better than not testing at all. In this sense, testing should be thought of as a practical tool for eliminating major problems and finding appropriate solutions for them. However, “Testing does not guarantee success or even prove that a product will be usable.” (p.26) The environment is always somewhat artificial and never the ideal place or circumstance of eventual use. Also, results of a test will depend greatly on the manner in which a test is conducted. This is why choosing an experienced moderator is crucial and making sure you have the right participants will also affect the validity of your data. Testing participants is one way of revealing usability problems, but different products call for different evaluation methods, some empirical and some non-empirical.

“When Should You Test?”

Based on the stage of development of your product, the authors suggest 4 corresponding types of tests, each focused on revealing different types of data in terms of your specific goals and concerns: exploratory (early stage), assessment (early and middle stages), validation (near-final stage), and comparison (all stages). When used as a complement to an iterative design process, those tests help ‘ “shape” the product to fit the end users’ abilities, expectations, and aptitude.’ (p.28)

The Exploratory or Formative Study is used at the early stage of the development process of your product. Here, testing will allow a high-level evaluation (horizontal functions) of the core concepts and ideas by revealing qualitative data. This stage also helps you reconsider your assumptions of your end users and understand them better in terms of their abilities to comprehend the visual language of an interface or product, their expectations of what a product does and how it functions, and their aptitude to learn and perform.

The Assessment or Summative Test is used early on and/or midway in the development process of your product. This test is a low-level evaluation (vertical functions) of the key tasks involved in using the product. How successful are the participants at completing at task? is typically your testing purpose and results in a quantitative performance data collection.

The Validation or Verification Test is used late in the process. The objective of this study is to define a appropriate ‘usability standard’ for future products and verify the efficiency and effectiveness of your product based on your two previous test results. Here, the test looks at various quantitative measurements such as: How well do users perform? How much time does it take them to complete a task? How consistent is the overall architecture of your design?

Finally, the Comparison Test usually comes in two forms: evaluating similar conceptual designs with slight variations or evaluating disparate alternatives of the same content. This method can be employed at any stage of the development process: to compare and explore different design layouts and visual (graphical or textual) languages used, and to determine the stance of your product against the competition.

 

related articles

Frog’s Research Learning Spiral (How the Why of data wins over the What)

Category : Designing for Usability, Interface Research, Research in Design Practice

As clients are becoming increasingly aware of ethnographic-ish research being an essential and integral part of the design process for a high quality user experience offer, it is important for us (designers) to learn an appropriate format for approaching and customizing research on a project basis.

Frog’s Research Learning Spiral, as David Sherwin names it in A Five-Step Process for Conducting User Research, allows us to think of research as not only a set of insight-focused methodologies and collaborative practices, but also very much so as a process of articulating and defining the focus area and scope of the research itself through its 5 learning stages: Objectives, Hypotheses, Methods, Conduct, and Synthesis.

I would like to focus on the the 3 early stages of the spiral. These are fundamental in situating the research area and addressing design questions with methodologies geared towards feeding our knowledge lexicon of people and things in their habitual contexts according to pre-defined objectives and hypotheses.

1 • Objectives focus on the framing of questions following the 3 Ws and an H structure: Who What When Where Why and How. These together help us define who the demographic user base is, what activities they might be involved in while using our service or product, when they would be engaged in such activities, where these activities would take place, under which emotional or rational states (why), and using which processes. These questions are in turn reformulated into simple statements of research objectives, which outline the scope of the research effort.

2 • Hypotheses are assumption packed opinions or suppositions we have about a product or service, its users, and the contextual settings in which the product acts, which are meant to be tested and challenged. Sherwin lists 3 types of hypotheses: attitude (what users would like to get out of a service), behavior (what users would like to do with the service), and feature (which feature users would most enjoy using).

3 • Methods — such as contextual enquiry, surveys, interviews, and benchmarking — can help prove or disprove these hypotheses by revealing key data about a demographic user, their contextual environments, and identify leverage points wherein design can affect their everyday and provide positive change or support. Other more participatory activities which involve probing users — such as diary studies, card sorting, and paper prototyping — can serve as experiential idea generating methods with a capacity for drawing design solutions and concepts that meet the user’s needs and mental models. Finally, evaluative methods — such as usability testing, heuristic evaluations, and cognitive walkthroughs — will demonstrate whether these ideas are effective, useful, and desirable.

[F]inding meaning in your data […] can mean reading between the lines and not taking a quote or something observed at face value. The why behind a piece of data is always more important than the what.” — Lauren Serota, Associate Creative Director at Frog Design

According to Sherwin, data tells us what and when users do things, but not why. Context is in fact king. Integrating such framework for user research helps provide us with the contextual understanding — the understanding of given demographics’ everydayness — for making more informed design decisions.

I am particularly interested in the name “learning spiral”: a looping process that doesn’t need to be lengthy, costly, and a unique event. It is spiral and has the potential of being a cyclical and iterative process, which can be applied as needed at different stages of a design process and with a different scope. That spiral from which I can learn allows me to investigate more specific areas of my users’ everyday by defining learning objectives.

While every research endeavour has a plan and objectives, i particularly found this interpretation because it gives importance to the planning and framing of research and integrates objectives definition as part of the research itself. Typically research seems to begin with contextual inquiry and interviewing right away as a recourse to inform the design approach and concept which does not necessarily end up being a solution that is desirable. Involving research participants in the framing of the research seems to be a more inclusive and humane approach that is bound to have a worthwhile and desirable quality.

— source: http://bit.ly/1eCoUyH

“A Human-Centered Technology.” – Don Norman

Category : Designing for Usability

When we speak of technology what we are often referring to are communication platforms, such as computers and cellular phones. For Don Norman (1993), anything that is created by human beings is a technological artifact, whether physical (tangible objects, i.e. paper) or mental (information structures, i.e. language). Norman divides technological innovations into good and bad, those that “make us smart” by extending human capabilities and those that “make us stupid” by frustrating, captivating, enslaving. His focus involves the understanding of both technologies that extend cognition and those that manipulate cognition.

While innovation has a historical contribution in terms of providing appropriate and efficient tools for society that help enrich human knowledge and enhance mental capabilities (memory, thought, reflection), it has however created modes of entertainment that increasingly promote a consuming rather than a productive people and is accountable for the apparent social divider between the haves and havenots of technology.

Moreover, new technologies create new social practices, specializations and knowledge requirements. This in turn adds a new set of social dividers between those who possess the required skills and those who don’t. The societal implications of new technologies have become more and more complex with the advent of computational systems. Norman’s aim is as follows: “If we learn the reasons for and the properties of these various technologies, then maybe we can control the impact.” His aim is to influence a human-centered approach to technology; a quest for technologies that demonstrate humane properties.

“Toward a Human-Centered View of Technology and People”

As technology becomes increasingly immaterial, its function concerns more and more the fulfillment of abstract needs, such as time-management, decision-making, methodologies, etc. Norman’s critique is that rather than aiding human cognition, technologies today add confusion. He notes that “distractibility”, “illogical” behaviour, and other concurrent characterizations are responses to the rigidness of the way in which computers operate. Those characterizations are related to the constant need for technology to impose a certain type of behavioral and mental effort (i.e. to pay attention, to speak grammatically, etc.). In addition, the information overload creates a feeling of inadequacy in society; such as, the inadequacy of the mind the memorize a certain amount of information, the incapacity to cope with innovation, etc. His premise is that “people err,” and computers don’t, instead they are faulty and the blame is made on the human.

‘To say that people often act illogically is to say that their behavior doesn’t fit the artificial mathematics called “logic” and “decision theory.” ‘

Hence, this adds yet another social divide: machine-centered behavior and human-centered behavior. For Norman, the problem lies in the way technology inadequacy to adapt to the human. Technology would benefit society if it understood more the way in which human cognition works. Norman also criticizes technology for creating an information saturation wherein humans are exposed to too much information that results in a so-called mental “burnout”. To illustrate this side-effect, Norman distinguishes between what technology is able to measure (i.e. hours worked per week) and what it can’t (i.e. satisfaction, rupture, experience), which he respectively refers to as “hard sciences” and ‘ “soft” sciences.’ The “hard sciences” make simple and objective measurements, leaving out what it cannot measure; whereas the ‘ “soft” sciences’ concerns itself with the complexity of subjective measurements, thereby making a link between technology and the living system.

“Two Kinds of Cognition”

The author identifies two kinds of cognition: experiential and reflective. Experiential cognition refers to the way we acquire expert skills through the use of our bodily skills; reflective cognition refers to mental thought processes that inform the development of new concepts and appropriate structures that serve as guides for action. Those are complementary and essential in the sense that they include both inner (mental) and outward (physical) practices.

The point here is that technologies that excite the mind (such as, TV) cannot be a substitute for experience, and hence cannot provide adequate modes of cognitive enhancement. “Experiential thought” is only possible through an active participation and engagement of the world.

Source:
Norman, Don. “A Human-Centered Technology.” Things That Make Us Smart: Defending Human Attributes in the Age of the Machine. Perseus Books, U.S.A., 1993.

@