Part 3: Be careful what you ask for: Designing Tasks for Quantitative Studies

In Part 1, we discussed the fundamentals of good task writing that apply to any kind of usability study. 

In Part 2, we discussed the right way to write tasks for qualitative studies.  Now let’s look at how to design tasks for quantitative studies.

When you undertake a quantitative study, you are looking for metrics that will help you understand patterns of behavior, spot trends, and draw reliable conclusions about how things perform.

Quant studies resemble scientific experiments, with rigorous, well-controlled conditions that enable you to capture metrics such as success, time on task, etc.

The goal of quant studies is numbers.

The primary goal of quant studies is to obtain numbers, not insights.  Your reports will contain lots of data visualizations such as bar charts, trend lines, and comparison tables.

In order to get clean data, you need to be very careful with the way you design the tasks you will have your testers do, and pay attention to way you administer the tests for every participant in your study.  Otherwise, you will introduce variables into the tests that will make the data unreliable.

Designing Good Tasks for Quantitative Studies

Here are some guidelines that will ensure success in your quant studies.

Make sure there’s only one way to do a task

Quant tasks must have only one possible interpretation and solution. If instructions are too broadly worded, participants may take different paths through the UI, and the metrics you get will be meaningless.  For example, if you are  timing them to see how long it takes to achieve success, it is important that you have a definite, singular, success goal for the task so you can stop the clock when they get there.  What is “success” when they have a lot of different viable paths they can take?

Bad: Find a vacation destination that fits your needs.

Better:  Book a flight to Papeete, Tahiti for two adults, for June.  

Provide enough details to keep the task narrow and focused

It isn’t enough to tell users which hotel to book at which time period — you also need to tell them what kind of room to book. This would not be required in a qual study, but it’s necessary in a quant test to make sure everyone will perform exactly the same task.  This is another way to ensure that you have a definitive success goal for the task.  We have found that when testers are performing tests that require them to work in hypothetical scenarios, they sometimes spend a lot of time coming up with “play-acting” responses.   If you have defined the task details for them, they won’t have to do that.  That will make your success time metrics much cleaner.

This even includes niceties like giving everyone the same login ID and password if you are testing something that requires them to authenticate.  If they don’t have to come up with a fake login, they’ll all take roughly the same amount of time for that step.

Remember the goal is numbers. You want to get results that let you report things like:

“Users were 29% faster at Task 1 when using Version A.”

So remove any wiggle-room from your tasks that will make the data squirrely.

Write tasks that can stand alone

It is a good practice to randomize the order of tasks in quantitative studies, to adjust for learning bias.  But you can’t do that if you write two tasks that build on each other, since those tasks must always be presented in the same order.  And if someone fails the first task they will automatically fail the second.

Bad:

Task 1: Find the personnel file on Abby Hayes.

Task 2: On Abby’s personnel file page, find her direct manager.

Better:

Task 1: Find the personnel file on Abby Hayes.

Task 2: On the page provided at this link, locate the direct manager of Mary McCafferty. 

Have only one success criterion per task

Don’t combine two tasks into one. In quant studies, you need a single, clear end point to help you determine time on task and whether the user was successful.

Bad:

Task 1: Find the online store and daily deal.

Better:

Task 1: Find the online store.

Task 2: Find the daily deal.

Once you have begun the study, don’t change the tasks

Each participant must do the exact same task in quantitative studies. These studies should hold all conditions constant except your independent variable (for example, your site vs. your competitor’s site).  So you don’t want to get part way through your tests and come up with a better question or a better way to ask something.  Any changes will contaminate the results — or at least make it impossible to aggregate the data from all of the testers.

Because of that, you really need to do pilot testing before any quant study.  You should recruit a couple of extra testers (exactly the same profile as your other testers) to use as your pilot testers and be prepared to throw away their results if you make improvements to your test after the pilot test.  It is money and time well spent, because of the data quality you will gain.

Focus on high priority tasks

Quant studies are expensive, and you will need to justify the costs to your stakeholders.  If you are in the habit of testing tasks that aren’t high priority for your users and your organization, you may make stakeholders less inclined to approve requests for budget.   But if you make it a practice to focus your efforts on the parts of the UI that are vital to the business —  lead-gen, ecommerce, business development, and customer retention — you will be able to demonstrate that all of your tests contribute to UI improvements that matter to the business.

With your qual tests you may have more flexibility to try tasks that are realistic edge cases, but your quant studies must focus on the most important, core tasks.

Practice Makes Perfect

It will take some deliberate effort and practice to get to the point where you instinctively know how to write tasks for qualitative and quantitative studies. Here’s a handy cheat sheet listing the guidelines for quantitative and qualitative studies.

Download Cheat Sheet PDF : Cheat Sheet- Careful

Next Installment

In Part 4,  I will explain how to ask good questions in your post-test interviews after qualitative and quantitative tests, as well as how to answer their questions during the test sessions.   Your choice of words can affect their answers and their behavior, and ultimately influence the outcome of the tests if you aren’t careful.