Usability test

The usability test is the most important part of the design of user-friendly devices because this is where you discover the problems.

The principle of a usability test is very simple: Tell various people to try using the product and observe them trying to make it work. Write down all the problems they encounter. Remember that the purpose of this test is not to prove that the thing works, but to find errors or problems.

You can apply usability tests to any kind of hardware - a can opener, a clock radio, or an airplane, and any type of software - a web page, a video game, a word processor, or a scientific program.

The type of errors you may find are:

The user has problems finding out how to use the product.
The user wants to do something that the product cannot do.
The product doesn't do what the user expects it to.
The user fails to discover useful features.
The user uses the product in an awkward way.
The user gets tired or has ergonomic problems .
Obviously, you may also discover plain old functional errors.

There are many ways to make a usability test. The most common methods are:

Interview the users.
Tell the users to think loud while they try to find out how to use the apparatus.
Watch the users over their shoulders.
Leave the users alone and watch them over closed-circuit television.

The last method may require a special set-up or laboratory, while the other tests can be carried out in the field, i.e. in a natural environment for using the product. The TV method may be useful for convincing others that the product has usability problems: The only thing that can convince your old boss that people can't find out how to use the gadget is to show him a video of a user making mistakes. Otherwise, a field test may be more realistic and easier to do.

Test persons and observers

The result of a usability test depends very much on the type of test persons used. Some examples of test user types are:

Novice. This is the person who has no prior experience with this type of apparatus. This type of test person will have the most problems finding out how to use the thing, and hence find many usability errors.

Experienced user. This is a person who has a lot of experience using a similar product. The experienced user will try the advanced features and will know where to look for problems.

Old inexperienced person. Old people learn more slowly and their senses and motor skills are less efficient. For example, my old mom has difficulties double-clicking a mouse because the mouse doesn't tolerate even the smallest movement between the two clicks.

Handicapped user. Letting people with various handicaps test your device can be quite revealing. If test persons with the right handicaps are not easily found then you may study various guidelines for making things accessible to handicapped persons. See the list of references.

Child. Children are curious and adventurous. They want to try everything and may push your device to the limits.

The noble and upright person. This is the type of person who will read the entire manual, including legal disclaimers, before daring to touch the ON button. This is the only person who will find errors in the printed manual, but they will never find out if your program can generate error messages.

The progressive and enthusiastic young man. He will try all the fancy features except the Help button. Tell him to find errors and he will consider it a game to defeat your gizmo.

The last two categories of test persons are well illustrated by the following example from my experience: I wanted to test the program shown below which draws the curves of mathematical functions.

Usability math software function experimenter

My first test person was a serene elderly engineer. I expected him to be proficient in math. He wrote y = x + 1 in the formula field and pressed Enter. The program showed a straight line. He then wrote y = x + 2. The program showed another straight line parallel to the first one. When he had got to y = x + 8 he stopped testing and said: It works!

The next test person I found was a young technician who suffered from a bipolar psychosis. He wrote all kinds of combinations of numbers and symbols without any regard to syntax. Silly as this may seem, it was an excellent opportunity to test if the program could generate useful error messages. At one point he got tired and wrote "I would rather have a beer!" in the formula field. He caught me on that one: I had never thought of what the program should do if the user types plain text (for example a question) instead of a formula.

Of course this extraordinary test person tried all buttons and menus in my program. Most of these gave an immediate response, but one complicated root-finding command took 30 seconds to complete. He clicked on this command, and the cursor turned into an hourglass to indicate that the program was working. He ignored this and clicked again and again impatiently. After he had clicked this time-consuming command ten times, frustrated by the lack of immediate response, he tried all the other buttons but got no response. The explanation was that the system served all commands on a first-in-first-out basis. The ten clicks for the time-consuming command were all put into a message queue so that the program was unable to do anything else for the next five minutes. A severe usability error!

I had to realize that this crazy man found more usability errors than all my other test persons together.

This shows how important it is to have more than one test person. No single test person will find all usability problems in your device. The more test persons you have the more problems you will find.

Interestingly, the problems found depend not only on the test person, but also on the observer. Laboratory experiments have shown that the problems found in a usability test depend almost as much on the observer as on the test person. A good test should therefore involve more than one observer as well as more than one test person.

You may even use different types of observers:

The designer. A person who has been involved in the technical construction of the device will observe if the user doesn't behave as expected, but may be inclined to think that the user is at fault rather than the device.

Another user. A person, who has as little knowledge of the internal technical structure as the test person has, can better understand the test person's problems and way of thought.

A usability expert. This type of observer has experience in noticing the test person's problems and frustrations but may not be familiar with the problems specific to this field of application.

Stages in the development

Remember that the attempt to solve one problem may create another problem so that you may have to test again. The complete development process for a new product may involve usability tests at several stages in the process:

Old version. Before starting to develop a new product you may test a similar preexisting product to see what needs to be improved.

Prototype. As soon as you have a mock-up or prototype you may make a usability test. Even a drawing of the user interface on a piece of paper can be used for a primitive test.

Beta version. A beta test may include the aspect of usability.

Release version. The final product should always be tested.

Feed back from customers

No usability test can find all problems because the test situation can never include all the situations that may occur in the real life use of a product. The next page describes how you can use feed-back from the end users to find usability problems.

Example

This is the tap in my bathroom. I bought it because it seemed just perfect: The right handle adjusts the temperature while the left handle adjusts the amount of water. Turn the left handle downwards if you want water out of the spout for washing your hands, or turn it upwards if you want a shower. Can you spot any usability problems in this tap?

Well, I couldn't until I had installed it. The first time I opened it to wash my hands I got water in my head. Why? Because I am habituated to the fact that you turn a tap counterclockwise to open. On this handle, counterclockwise is up which means shower. It took me a long time to get used to turning the handle the other way, and I still do it wrong sometimes. And every time I have guests and they need to go to the toilet, they come out very angry and wet all over!

This is the type of error that you don't find unless you make a usability test.

← Involve users in the design · Feedback from customers →