Barker - Chapter 8 - Conducting Usability Tests

Writing Software Documentation: A Task-Oriented Approach

Thomas T. Barker

Chapter 8 – Conducting Usability Tests

Barker defines the process of testing documents as “procedures for gaining empirical data about [their] usability.” This is accomplished using three basic types of tests: procedure tests for task performance, tutorial tests for skill and understanding, and reference tests for access to information. The chapter presents a 10-step test plan for conducting these tests.

Guidelines

Good testing requires careful planning. Barker suggests the following 10 steps be followed.

Step 1 – Decide When to Test

Testing often occurs after a draft of the documents has been produced, however it can take place at various stages in the project depending on your testing goals. A Predictive Test can be conducted during the design stage, a Remedial Test during writing or development, and an Evaluative Test after the document has been completed and delivered.

You must also determine what parts of your documentation you want to test. Determining this can be assisted by relying on the objectives that you set for your documentation, and the user analysis that you previously performed.

Step 2 – Select the Test Points

Test points are issues or features that can interfere with the efficient and effective application of a program to a user’s work activities. These can include problems with content and problems with document design.

Try to identify points where a mistake on either of these levels could lead to user failure. Any procedure where the cost of user confusion or failure is high in terms of time or money would be a good candidate for testing. In addition to testing these procedures, you should also test the document design strategies employed to ensure that they are effective.

Step 3 – Choose the Type of Test

There are three types of tests. Performance Tests test whether users can successfully complete a given procedure. Understandability Tests test whether users can provide evidence of what they have learned. Read-and-Locate Tests test how effectively users can locate a given topic of information in a documentation set.

Step 4 – Set Performance and Learning Objectives

Performance objectives state how long a procedure should take or the frequency of correctness one can expect from users. They can be time-related, or error-related. The goal here is to collect numbers that can be analyzed and compared.

In order to produce objective test results, it is important to avoid letting the outcome of the test be skewed by bias. Prejudices can creep in as a result of various work pressures, a proforma testing atmosphere, or becoming too enamored with the product of your labors.

Step 5 – Select Testers and Evaluators

The tester is defined as the person who administers the test. The evaluator is the person actually taking the usability test. If you do not have actual users available, you will have to compromise by selecting individuals to stand in for users.

Step 6 – Prepare the Test Materials

The written and other materials provided to testers and evaluators can be very extensive. Barker lists multiple forms of test materials, both written and hardware/software based. He also discusses the importance of pilot testing, which basically amounts to “testing the test”, to ensure that the instructions, terminology, and time expectations are appropriate.

Step 7 – Set Up the Test Environment

The test environment can range from the user’s actual workplace to a controlled laboratory. A field test produces valuable real-world insight, while a testing lab offers greater control over the process. A combination of both environments is often the most effective approach.

Step 8 – Record Information Accurately

It is important to accurately record what you see and hear throughout the test. This can be accomplished using video and audio recording technology, in conjunction with copious note taking. Making use of additional observers can help fill in any details that you miss.

Step 9 – Interpret the Data

There are a number of phenomenon that can cloud the data obtained through testing. Once these variables have been accounted for, you can make changes to your document based on what the numbers reveal. These can inspire all sorts of design decisions that require skill and commonsense decisions on the part of the author.

Step 10 – Incorporate the Feedback

The final step is an obvious one. The information gleaned from the testing process should serve to enhance the document in a positive way.

Discussion

In this section Barker offers a variety of thoughts on the usability testing of technical documentation. These include topics such as:

• The three components of usability testing: tester, evaluator, and subject matter.
  
• The importance of user testing as part of user-driven design
  
• Comparing field tests and laboratory tests
  
• The cost of testing and making it a corporate priority
  
• Various field testing methods
  
• The trade offs between testing earlier and testing later
  
• Distinguishing between problems with the documentation and problems with the product

Barker concludes the chapter with a glossary of terms and a checklist for use when planning and executing usability tests.